Merge branch 'code/majorrig/wiki-hevc-already-failed-skip'

04-streaming/plex/hevc-vaapi-batch-encode.md

@@ -5,7 +5,7 @@ category: plex
 tags: [plex, ffmpeg, hevc, vaapi, amd, gpu, encode, storage, rx480]
 status: published
 created: 2026-05-15
-updated: 2026-05-22
+updated: 2026-06-05
 ---
 # HEVC Batch Re-Encode for Plex Using VAAPI (AMD GPU)
 
@@ -121,7 +121,7 @@ Each file logs:
 
 ### Space guard
 
-The script aborts if free space on the Plex volume drops below 20GB (`MIN_FREE_GB`). Worst-case headroom needed is `source_size + tmp_size` simultaneously — on a 4GB source file that's ~8GB peak.
+The script aborts if free space on the Plex volume drops below 10GB (`MIN_FREE_GB`). Worst-case headroom needed is `source_size + tmp_size` simultaneously — on a 4GB source file that's ~8GB peak. Note: the space check only runs at the **start** of each encode, not during — a large file can still consume significant disk mid-encode.
 
 ---
 
@@ -278,3 +278,54 @@ local tmp="${dir}/${safe_stem}.hevc.tmp.${ext}"
 
 After patching, delete the affected entries from `hevc_failed.txt` (or leave them — they'll be re-queued on the next run since they're not in `hevc_done.txt`) and restart the batch.
 
+---
+
+### Many files failing: output larger than source (streaming content)
+
+**Symptom:** A large portion of the queue ends up in `hevc_failed.txt` with log lines like:
+
+```
+[2026-06-05 ...] Output: 4.7G  savings=0 (output larger than source)
+[2026-06-05 ...] WARN: output is larger than source — skipping swap, keeping original
+```
+
+**Cause:** These files are YouTube downloads or streaming archives (Giant Bomb, Twitch VODs, etc.) that were already encoded with an efficient H.264 encoder (typically YouTube's VP9-to-AVC pipeline or a broadcast H.264 encoder at a reasonable bitrate). VAAPI HEVC encoding at QP 28 on a Polaris GPU (RX 480/580) is a hardware encoder with limited rate control precision — it cannot beat a well-tuned software H.264 encode on already-compressed talking-head/gaming content. The output reliably comes out 15–25% *larger* than the source.
+
+The script handles this correctly: it detects output > source, deletes the tmp, keeps the original, and writes to `hevc_failed.txt`. The files are not corrupted. However, without the `already_failed()` guard, the script will re-attempt these files on every queue rebuild, wasting CPU time and briefly consuming 4–8 GB of disk per failed attempt.
+
+**Fix — add `already_failed()` skip logic:**
+
+Patch `~/hevc_batch.sh` to skip files already in `hevc_failed.txt`:
+
+```bash
+# After the existing already_done() function, add:
+already_failed() {
+  [[ -f "$FAILED" ]] && grep -qF "$1" "$FAILED"
+}
+
+# In build_queue(), after the already_done "$f" && continue line:
+already_failed "$f" && continue
+
+# In the main loop, after the already_done "$file" check:
+already_failed "$file" && { log "SKIP (already failed): $file"; continue; }
+```
+
+After patching, the batch will skip all 132+ known-bad files on the next pass and only attempt fresh queue entries.
+
+**Tuning options to improve savings on dense content:**
+
+- Lower QP: `--qp 24` or `--qp 22` — more aggressive quality target, better chance of beating source size. Trade-off: larger output for files that do compress.
+- Accept the failures: for streaming content archives, the source is already "good enough." Only files that are genuinely oversized H.264 (old stream captures at very high bitrate) will benefit from HEVC re-encode.
+
+**Identifying which files are worth encoding:**
+
+```bash
+# Show source bitrate for all queued files — high-bitrate sources are candidates
+while IFS= read -r f; do
+  bitrate=$(ffprobe -v quiet -show_entries format=bit_rate -of csv=p=0 "$f" 2>/dev/null)
+  echo "$bitrate $f"
+done < ~/hevc_queue.txt | sort -rn | head -20
+```
+
+Files above ~8,000 kbits/s are typically good encode candidates. Files at 3,000–5,000 kbits/s (typical YouTube/Twitch 1080p) will usually fail.
+