Add 5 wiki articles from 2026-04-17/18 work

- ghost-smtp-mailgun-setup: two-system email config (newsletter API + transactional SMTP) - firewalld-fleet-hardening: Fedora fleet firewall audit-and-harden pattern with Ansible - clamav-fleet-deployment: fleet deployment with nice/ionice throttling + quarantine - ansible-check-mode-false-positives: when: not ansible_check_mode guard for verify/assert tasks - ghost-emailanalytics-lag-warning: submitted status, lag counter, fetchMissing skip explained
2026-04-18 11:13:39 -04:00
parent d616eb2afb
commit b40e484aae
45 changed files with 703 additions and 39 deletions
--- a/02-selfhosting/security/.keep
+++ b/02-selfhosting/security/.keep
--- a/02-selfhosting/security/clamav-fleet-deployment.md
+++ b/02-selfhosting/security/clamav-fleet-deployment.md
@@ -0,0 +1,154 @@
+---
+title: ClamAV Fleet Deployment with Ansible
+domain: selfhosting
+category: security
+tags:
+  - clamav
+  - antivirus
+  - security
+  - ansible
+  - fleet
+  - cron
+status: published
+created: 2026-04-18
+updated: 2026-04-18T11:13
+---
+# ClamAV Fleet Deployment with Ansible
+
+## Overview
+
+ClamAV is the standard open-source antivirus for Linux servers. For internet-facing hosts, a weekly scan with fresh definitions catches known malware, web shells, and suspicious files before they cause damage. The key operational concern is CPU impact — an unthrottled `clamscan` will saturate a core for hours on a busy host. The solution is `nice` and `ionice` wrappers.
+
+> This guide covers deployment to internet-facing hosts. Internal-only hosts (storage, inference, gaming) are lower priority and can be skipped.
+
+## What Gets Deployed
+
+- `clamav` + `clamav-update` packages (provides `clamscan` + `freshclam`)
+- `freshclam` service enabled for automatic definition updates
+- A quarantine directory at `/var/lib/clamav/quarantine/`
+- A weekly `clamscan` cron job, niced to background priority
+- SELinux context set on the quarantine directory (Fedora hosts)
+
+## Ansible Playbook
+
+```yaml
+- name: Deploy ClamAV to internet-facing hosts
+  hosts: internet_facing  # dca, majorlinux, teelia, tttpod, majortoot, majormail
+  become: true
+
+  tasks:
+
+    - name: Install ClamAV packages
+      ansible.builtin.package:
+        name:
+          - clamav
+          - clamav-update
+        state: present
+
+    - name: Enable and start freshclam
+      ansible.builtin.service:
+        name: clamav-freshclam
+        enabled: true
+        state: started
+
+    - name: Create quarantine directory
+      ansible.builtin.file:
+        path: /var/lib/clamav/quarantine
+        state: directory
+        owner: root
+        group: root
+        mode: '0700'
+
+    - name: Set SELinux context on quarantine dir (Fedora/RHEL)
+      ansible.builtin.command:
+        cmd: chcon -t var_t /var/lib/clamav/quarantine
+      when: ansible_os_family == "RedHat"
+      changed_when: false
+
+    - name: Deploy weekly clamscan cron job
+      ansible.builtin.cron:
+        name: "Weekly ClamAV scan"
+        user: root
+        weekday: "0"   # Sunday
+        hour: "3"
+        minute: "0"
+        job: >-
+          nice -n 19 ionice -c 3
+          clamscan -r /
+          --exclude-dir=^/proc
+          --exclude-dir=^/sys
+          --exclude-dir=^/dev
+          --exclude-dir=^/run
+          --move=/var/lib/clamav/quarantine
+          --log=/var/log/clamav/scan.log
+          --quiet
+          2>&1 | logger -t clamscan
+```
+
+## The nice/ionice Flags
+
+Without throttling, `clamscan -r /` will peg a CPU core for 30–90 minutes depending on disk size and file count. On production hosts this causes Netdata alerts and visible service degradation.
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `nice -n 19` | Lowest CPU priority | Kernel will preempt this process for anything else |
+| `ionice -c 3` | Idle I/O class | Disk I/O only runs when no other process needs the disk |
+
+With both flags set, `clamscan` becomes essentially invisible under normal load. The scan takes longer (possibly 2–4× on busy disks), but this is acceptable for a weekly background job.
+
+> **SELinux on Fedora/Fedora:** `ionice` may trigger AVC denials under SELinux Enforcing. If scans silently fail on Fedora hosts, check `ausearch -m avc -ts recent` for `clamscan` denials. See [selinux-fail2ban-execmem-fix](../../05-troubleshooting/selinux-fail2ban-execmem-fix.md) for the pattern.
+
+## Excluded Paths
+
+Always exclude virtual/pseudo filesystems — scanning them wastes time and can trigger false positives or kernel errors:
+
+```
+--exclude-dir=^/proc   # Process info (not real files)
+--exclude-dir=^/sys    # Kernel interfaces
+--exclude-dir=^/dev    # Device nodes
+--exclude-dir=^/run    # Runtime tmpfs
+```
+
+You may also want to exclude large data directories (`/var/lib/docker`, backup volumes, media stores) if scan time is a concern. These are lower-risk targets anyway.
+
+## Quarantine vs Delete
+
+`--move=/var/lib/clamav/quarantine` moves detected files rather than deleting them. This is safer than `--remove` — you can inspect and restore false positives. Review the quarantine directory periodically:
+
+```bash
+ls -la /var/lib/clamav/quarantine/
+```
+
+If a file is a confirmed false positive, restore it and add it to `/etc/clamav/whitelist.ign2`.
+
+## Checking Scan Results
+
+```bash
+# View last scan log
+cat /var/log/clamav/scan.log
+
+# Summary line from the log
+grep -E "^Infected|^Scanned" /var/log/clamav/scan.log | tail -5
+
+# Check freshclam is keeping definitions current
+systemctl status clamav-freshclam
+freshclam --version
+```
+
+## Verifying Deployment
+
+Test that ClamAV can detect malware using the EICAR test file (a harmless string that all AV tools recognize as test malware):
+
+```bash
+echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' \
+  > /tmp/eicar-test.txt
+clamscan /tmp/eicar-test.txt
+# Expected: /tmp/eicar-test.txt: Eicar-Signature FOUND
+rm /tmp/eicar-test.txt
+```
+
+## See Also
+
+- [clamscan-cpu-spike-nice-ionice](../../05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — troubleshooting CPU spikes from unthrottled scans
+- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)
+- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md)
--- a/02-selfhosting/security/firewalld-fleet-hardening.md
+++ b/02-selfhosting/security/firewalld-fleet-hardening.md
@@ -0,0 +1,173 @@
+---
+title: Firewall Hardening with firewalld on Fedora Fleet
+domain: selfhosting
+category: security
+tags:
+  - firewall
+  - firewalld
+  - iptables
+  - fedora
+  - ansible
+  - security
+  - hardening
+status: published
+created: 2026-04-18
+updated: 2026-04-18T11:13
+---
+# Firewall Hardening with firewalld on Fedora Fleet
+
+## Overview
+
+Fedora and RHEL-based hosts use `firewalld` as the default firewall manager, backed by `nftables`. Over time, firewall rules accumulate stale entries — decommissioned services, old IP allowances, leftover port forwards — that widen the attack surface silently. This article covers the audit-and-harden pattern for Fedora fleet hosts using Ansible.
+
+> For Ubuntu/Debian hosts, see [ufw-firewall-management](ufw-firewall-management.md).
+
+## The Problem with Accumulated Rules
+
+Rules added manually or by service installers (`firewall-cmd --add-port=...`) don't get cleaned up when services are removed. Common sources of stale rules:
+
+- Monitoring agents (Zabbix, old Netdata exporters)
+- Media servers moved to another host (Jellyfin, Plex)
+- Development ports left open during testing
+- IP-specific allowances for home IPs that have since changed
+
+These stale rules are invisible in day-to-day operation but show up during audits as unnecessary exposure.
+
+## Auditing Current Rules
+
+```bash
+# Show all active rules (nftables, what firewalld actually uses)
+nft list ruleset
+
+# Show firewalld zones and services
+firewall-cmd --list-all-zones
+
+# Show permanent config (what survives reboot)
+firewall-cmd --permanent --list-all
+```
+
+Cross-reference open ports against running services:
+
+```bash
+# What's actually listening?
+ss -tlnp
+
+# Match against firewall rules — anything open that has no listener is stale
+```
+
+## Ansible Hardening Approach
+
+Rather than patching rules incrementally, the cleanest approach is to **flush and rebuild**: remove all non-essential rules and explicitly whitelist only what the host legitimately serves. This avoids drift and makes the resulting ruleset self-documenting.
+
+The Ansible playbook uses `ansible.posix.firewalld` to manage rules declaratively and a flush task to clear the slate before applying the desired state.
+
+### Pattern: Flush → Rebuild
+
+```yaml
+- name: Remove stale firewalld rules
+  ansible.posix.firewalld:
+    port: "{{ item }}"
+    permanent: true
+    state: disabled
+  loop:
+    - 8096/tcp   # Jellyfin — decommissioned
+    - 10050/tcp  # Zabbix agent — removed
+    - 10051/tcp  # Zabbix server — removed
+  ignore_errors: true  # OK if rule doesn't exist
+
+- name: Apply minimal whitelist
+  ansible.posix.firewalld:
+    port: "{{ item }}"
+    permanent: true
+    state: enabled
+  loop: "{{ allowed_ports }}"
+  notify: Reload firewalld
+```
+
+Define `allowed_ports` per host in `host_vars/`:
+
+```yaml
+# host_vars/majorlab/firewall.yml
+allowed_ports:
+  - 80/tcp    # Caddy HTTP
+  - 443/tcp   # Caddy HTTPS
+  - 22/tcp    # SSH (public)
+  - 2222/tcp  # SSH (alt)
+  - 3478/tcp  # Nextcloud Talk TURN
+```
+
+### Tailscale SSH: Restrict to ts-input Zone
+
+For hosts where SSH should only be accessible via Tailscale, move the SSH rule from the public zone to the `ts-input` interface:
+
+```yaml
+- name: Remove SSH from public zone
+  ansible.posix.firewalld:
+    zone: public
+    service: ssh
+    permanent: true
+    state: disabled
+
+- name: Allow SSH on Tailscale interface only
+  ansible.posix.firewalld:
+    zone: trusted
+    interface: tailscale0
+    permanent: true
+    state: enabled
+  notify: Reload firewalld
+```
+
+> **Note:** The Tailscale interface is `tailscale0` unless customized. Confirm with `ip link show` before applying.
+
+## Per-Host Hardening Reference
+
+Different host roles need different rule sets. These are the minimal whitelists for common MajorsHouse host types:
+
+| Host Role | Open Ports | Notes |
+|-----------|-----------|-------|
+| Reverse proxy (Caddy) | 80, 443, 22/2222 | No app ports exposed — Caddy proxies internally |
+| Storage/media (Plex) | 32400 (public), 22 (Tailscale-only) | Plex needs public; SSH Tailscale-only |
+| Bot/Discord host | 25 (Postfix), 25000 (webUI), 6514 (syslog-TLS) | No inbound SSH needed if Tailscale-only |
+| Mail server | 25, 587, 993, 443, 22 | Standard mail ports |
+
+## Default Policy
+
+Set the default zone policy to `DROP` (not `REJECT`) to make the host non-discoverable:
+
+```bash
+firewall-cmd --set-default-zone=drop --permanent
+firewall-cmd --reload
+```
+
+`DROP` silently discards packets; `REJECT` sends an ICMP unreachable back, confirming the host exists.
+
+## Verifying After Apply
+
+```bash
+# Confirm active rules match intent
+firewall-cmd --list-all
+
+# Spot-check a port that should be closed
+nmap -p 10050 <host-ip>
+# Expected: filtered (not open, not closed)
+
+# Confirm a port that should be open
+nmap -p 443 <host-ip>
+# Expected: open
+```
+
+## Ansible Handler
+
+```yaml
+handlers:
+  - name: Reload firewalld
+    ansible.builtin.service:
+      name: firewalld
+      state: reloaded
+```
+
+## See Also
+
+- [ufw-firewall-management](ufw-firewall-management.md) — Ubuntu/Debian equivalent
+- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md)
+- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)