From 64ac418a36d8b9f6b07ef26bf84d40f9f786de20 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 15 May 2026 09:02:24 -0400 Subject: [PATCH] wiki: add ClamAV daemonless mode section + HEVC VAAPI article link --- .../security/clamav-fleet-deployment.md | 37 +++- 04-streaming/plex/hevc-vaapi-batch-encode.md | 168 ++++++++++++++++++ SUMMARY.md | 3 +- 3 files changed, 206 insertions(+), 2 deletions(-) create mode 100644 04-streaming/plex/hevc-vaapi-batch-encode.md diff --git a/02-selfhosting/security/clamav-fleet-deployment.md b/02-selfhosting/security/clamav-fleet-deployment.md index 4e6134c..f4a2888 100644 --- a/02-selfhosting/security/clamav-fleet-deployment.md +++ b/02-selfhosting/security/clamav-fleet-deployment.md @@ -11,7 +11,7 @@ tags: - cron status: published created: 2026-04-18 -updated: 2026-05-10T01:50 +updated: 2026-05-15T03:00 --- # ClamAV Fleet Deployment with Ansible @@ -226,6 +226,41 @@ The "polite CPU is invisible to DO" trick stops working once the box is small en **Alternative considered: switch to `clamdscan`** — uses a resident `clamd` daemon, signatures stay loaded, scan finishes ~10× faster with much less CPU/RAM. Better long-term answer, but requires running `clamd` continuously (memory cost on small boxes is ~250 MB resident vs the cron approach which only holds RAM during scan). Trade-off, not strictly better. +## Daemonless Mode on Memory-Constrained Hosts + +On hosts with ≤2 GB RAM, running `clamd` continuously is often counterproductive. The daemon loads its full signature database (~950 MB RSS) into memory and keeps it resident. On small VMs this crowds out MySQL, PHP-FPM, and other services — often pushing the whole system into swap rather than preventing anything. + +**Affected hosts (fleet history):** + +| Host | RAM | Incident | Resolution | +|------|-----|----------|------------| +| teelia | 1.9 GB | 2026-04-27 — clamd 728 MB RSS, 94% RAM alert | daemonless | +| dcaprod | 3.8 GB | 2026-04-30 — clamd OOM thrash after 512M cgroup cap | daemonless | +| majorlinux | 2.0 GB | 2026-05-15 — clamd 980 MB swap, mysqld swapping 293 MB | daemonless | + +**The fix: `clamav_use_daemon: false` host_var** + +`configure_clamav.yml` supports a per-host override. Add to the host's `host_vars//vars.yml`: + +```yaml +clamav_use_daemon: false +``` + +Then re-run the playbook: + +```bash +ansible-playbook configure_clamav.yml --limit +``` + +This will: +- Stop and disable `clamav-daemon.service` and `clamav-daemon.socket` +- Deploy the weekly scan template using `clamscan` (daemonless, loads DB per run) +- Leave `clamav-freshclam` active so definitions stay current + +**Trade-off:** Each weekly scan loads the signature DB fresh (~950 MB peak RAM for the scan duration, then freed). The scan takes longer than `clamdscan` (~3–5× on a warm daemon), but this is acceptable for a weekly background job. The `systemd-run MemoryMax` cgroup wrapper in the scan template caps peak usage so the scan can't OOM the host. + +**Rule of thumb:** Use daemon mode (`clamav_use_daemon: true` or unset) on hosts with ≥4 GB RAM where scan speed matters (mail servers, upload handlers). Use daemonless on webservers and small VMs where continuous memory residency is the bigger risk. + ## See Also - [clamscan-cpu-spike-nice-ionice](../../05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — troubleshooting CPU spikes from unthrottled scans diff --git a/04-streaming/plex/hevc-vaapi-batch-encode.md b/04-streaming/plex/hevc-vaapi-batch-encode.md new file mode 100644 index 0000000..0df3b47 --- /dev/null +++ b/04-streaming/plex/hevc-vaapi-batch-encode.md @@ -0,0 +1,168 @@ +--- +title: "HEVC Batch Re-Encode for Plex Using VAAPI (AMD GPU)" +domain: streaming +category: plex +tags: [plex, ffmpeg, hevc, vaapi, amd, gpu, encode, storage, rx480] +status: published +created: 2026-05-15 +updated: 2026-05-15 +--- +# HEVC Batch Re-Encode for Plex Using VAAPI (AMD GPU) + +## Problem + +Plex NVMe storage is filling up from a large library of H.264-encoded video files (YouTube downloads, stream archives, etc.). Re-encoding to HEVC (H.265) reclaims 30–50% of disk space. The catch: Plex tracks each file's "date added" in a SQLite database, and that order matters for playback queues. Naive re-encode-and-replace approaches can corrupt or reset that metadata. + +## Solution + +Use `ffmpeg` with `hevc_vaapi` (AMD GPU hardware encoder) to batch re-encode files in-place using an atomic rename swap that preserves the Plex database record — including `added_at` — without any Plex downtime or database editing. + +--- + +## How Plex Stores "Date Added" + +Plex does **not** use file modification time (`mtime`) for "date added." It stores a Unix timestamp in its SQLite database: + +```sql +-- Plex DB location (override via systemd unit may differ — check): +-- /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/ +-- Plug-in Support/Databases/com.plexapp.plugins.library.db +-- (or wherever PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR points) + +SELECT mi.added_at, datetime(mi.added_at, 'unixepoch'), mp.file +FROM metadata_items mi +JOIN media_items me ON me.metadata_item_id = mi.id +JOIN media_parts mp ON mp.media_item_id = me.id +WHERE mp.file LIKE '%your-file%'; +``` + +> **Note:** If the default path returns 0 rows, check your actual data directory: +> ```bash +> systemctl cat plexmediaserver | grep APPLICATION_SUPPORT +> ``` + +The `added_at` field is keyed to the **file path** in `media_parts`. As long as the file path doesn't change, the database record — including `added_at` — is untouched even after the file's content is replaced. + +--- + +## Why VAAPI Instead of libx265 + +On a host with an AMD RX 480/580 (or similar Polaris GPU), hardware HEVC encoding via VAAPI is roughly **9× faster** than software libx265 at comparable quality: + +| Encoder | Speed (1080p) | Notes | +|---|---|---| +| libx265 -preset medium | ~21 fps / 0.35× | Best quality/size ratio | +| hevc_vaapi QP 28 | ~186 fps / 3.1× | Sufficient for streaming content | + +For 1080p streaming content (game streams, podcasts, YouTube archival), the quality difference is imperceptible. libx265 is preferable only for archival encodes where absolute quality matters. + +### Verify VAAPI is working + +```bash +vainfo 2>&1 | grep -E "vaapi|HEVC|hevc|Driver" +ls /dev/dri/renderD128 +``` + +You need `VAProfileHEVCMain : VAEntrypointEncSlice` in the output. If missing, install `mesa-va-drivers-freeworld` (RPM Fusion) for AMD hardware. + +--- + +## The Atomic Swap Strategy + +The key insight: `mv file.tmp file` on the **same filesystem** is an atomic inode rename at the kernel level. Plex sees the same path still present — it never fires a "file removed" event, so the `metadata_items` record (including `added_at`) is preserved. + +**Safe sequence:** +1. Encode source → `.hevc.tmp.mp4` alongside the original +2. Verify the output with `ffprobe` +3. `touch -r original.mp4 temp.mp4` — copy mtime (cosmetic, not required) +4. `mv temp.mp4 original.mp4` — atomic replace + +**The one pitfall:** if the original file is deleted *before* the `mv`, Plex orphans the DB record (removes `metadata_items` entry on next scan) and re-indexes the new file with a fresh `added_at`. The original must still exist at swap time. + +--- + +## The Batch Script + +Script lives at `~/hevc_batch.sh` on majorhome. + +```bash +# Dry run — scan and report what would be encoded, no changes +bash ~/hevc_batch.sh --dry-run + +# Full run (default: files >1GB, QP 28) +tmux new-session -d -s hevc_batch 'bash ~/hevc_batch.sh' + +# Custom options +bash ~/hevc_batch.sh --min-size-gb 2 --qp 26 +``` + +### Queue and resume + +The script writes a queue file at `~/hevc_queue.txt` on first run (scanning all files with ffprobe — takes ~10 min for a large library). On subsequent runs it resumes from where it left off. Completed files are logged to `~/hevc_done.txt`. Failed files go to `~/hevc_failed.txt`. + +To restart from scratch: `rm ~/hevc_queue.txt ~/hevc_done.txt` + +### Log output + +```bash +# Structured log lines only (skip ffmpeg progress noise) +grep '^\[20' ~/hevc_batch.log + +# Watch live progress +tail -f ~/hevc_batch.log | grep '^\[20' +``` + +Each file logs: +- Source size and codec +- `Plex added_at before: ` +- ffmpeg exit code and elapsed time +- Output size and savings +- `DB check: added_at PRESERVED ✓` (or WARN if changed) + +### Space guard + +The script aborts if free space on the Plex volume drops below 20GB (`MIN_FREE_GB`). Worst-case headroom needed is `source_size + tmp_size` simultaneously — on a 4GB source file that's ~8GB peak. + +--- + +## ffmpeg Command + +```bash +ffmpeg \ + -vaapi_device /dev/dri/renderD128 \ + -i "input.mp4" \ + -vf 'format=nv12,hwupload' \ + -c:v hevc_vaapi -rc_mode CQP -qp 28 \ + -c:a copy \ + -movflags +faststart \ + -y "output.tmp.mp4" +``` + +- `-rc_mode CQP -qp 28` — constant quantizer; higher value = smaller file / lower quality. QP 24 is high quality, QP 28 is good for streaming content. +- `-vf 'format=nv12,hwupload'` — required to move frames to GPU memory for VAAPI encoding. +- `-c:a copy` — passes audio through untouched. +- `hevc_vaapi` does not support 10-bit output on Polaris (RX 480/580). For 10-bit HDR sources, fall back to `libx265` with color signaling flags. + +--- + +## Plex Data Directory Override + +On majorhome, the Plex data directory is overridden in the systemd unit — the default path `/var/lib/plexmediaserver/` is empty: + +```bash +systemctl cat plexmediaserver | grep APPLICATION_SUPPORT +# Environment=PLEX_MEDIA_SERVER_APPLICATION_SUPPORT_DIR=/plex/plexdata/Library/Application Support +``` + +The actual DB path is therefore: +``` +/plex/plexdata/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db +``` + +--- + +## Related + +- [[plex-4k-codec-compatibility]] — Apple TV Direct Play compatibility, HEVC HDR notes +- [[snapraid-mergerfs-setup]] — MajorRAID storage pool setup +- [[SnapRAID-Majorhome]] — majorhome SnapRAID project diff --git a/SUMMARY.md b/SUMMARY.md index c098038..ed14416 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,6 @@ --- created: 2026-04-02T16:03 -updated: 2026-05-11T07:35 +updated: 2026-05-15T09:00 --- * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) @@ -70,6 +70,7 @@ updated: 2026-05-11T07:35 * [Streaming & Podcasting](04-streaming/index.md) * [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) * [Plex 4K Codec Compatibility (Apple TV)](04-streaming/plex/plex-4k-codec-compatibility.md) + * [HEVC Batch Re-Encode for Plex Using VAAPI (AMD GPU)](04-streaming/plex/hevc-vaapi-batch-encode.md) * [Troubleshooting](05-troubleshooting/index.md) * [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) * [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md)