Add 5 wiki articles from 2026-04-17/18 work

- ghost-smtp-mailgun-setup: two-system email config (newsletter API + transactional SMTP)
- firewalld-fleet-hardening: Fedora fleet firewall audit-and-harden pattern with Ansible
- clamav-fleet-deployment: fleet deployment with nice/ionice throttling + quarantine
- ansible-check-mode-false-positives: when: not ansible_check_mode guard for verify/assert tasks
- ghost-emailanalytics-lag-warning: submitted status, lag counter, fetchMissing skip explained

02-selfhosting/security/clamav-fleet-deployment.md

@@ -0,0 +1,154 @@
+---
+title: ClamAV Fleet Deployment with Ansible
+domain: selfhosting
+category: security
+tags:
+  - clamav
+  - antivirus
+  - security
+  - ansible
+  - fleet
+  - cron
+status: published
+created: 2026-04-18
+updated: 2026-04-18T11:13
+---
+# ClamAV Fleet Deployment with Ansible
+
+## Overview
+
+ClamAV is the standard open-source antivirus for Linux servers. For internet-facing hosts, a weekly scan with fresh definitions catches known malware, web shells, and suspicious files before they cause damage. The key operational concern is CPU impact — an unthrottled `clamscan` will saturate a core for hours on a busy host. The solution is `nice` and `ionice` wrappers.
+
+> This guide covers deployment to internet-facing hosts. Internal-only hosts (storage, inference, gaming) are lower priority and can be skipped.
+
+## What Gets Deployed
+
+- `clamav` + `clamav-update` packages (provides `clamscan` + `freshclam`)
+- `freshclam` service enabled for automatic definition updates
+- A quarantine directory at `/var/lib/clamav/quarantine/`
+- A weekly `clamscan` cron job, niced to background priority
+- SELinux context set on the quarantine directory (Fedora hosts)
+
+## Ansible Playbook
+
+```yaml
+- name: Deploy ClamAV to internet-facing hosts
+  hosts: internet_facing  # dca, majorlinux, teelia, tttpod, majortoot, majormail
+  become: true
+
+  tasks:
+
+    - name: Install ClamAV packages
+      ansible.builtin.package:
+        name:
+          - clamav
+          - clamav-update
+        state: present
+
+    - name: Enable and start freshclam
+      ansible.builtin.service:
+        name: clamav-freshclam
+        enabled: true
+        state: started
+
+    - name: Create quarantine directory
+      ansible.builtin.file:
+        path: /var/lib/clamav/quarantine
+        state: directory
+        owner: root
+        group: root
+        mode: '0700'
+
+    - name: Set SELinux context on quarantine dir (Fedora/RHEL)
+      ansible.builtin.command:
+        cmd: chcon -t var_t /var/lib/clamav/quarantine
+      when: ansible_os_family == "RedHat"
+      changed_when: false
+
+    - name: Deploy weekly clamscan cron job
+      ansible.builtin.cron:
+        name: "Weekly ClamAV scan"
+        user: root
+        weekday: "0"   # Sunday
+        hour: "3"
+        minute: "0"
+        job: >-
+          nice -n 19 ionice -c 3
+          clamscan -r /
+          --exclude-dir=^/proc
+          --exclude-dir=^/sys
+          --exclude-dir=^/dev
+          --exclude-dir=^/run
+          --move=/var/lib/clamav/quarantine
+          --log=/var/log/clamav/scan.log
+          --quiet
+          2>&1 | logger -t clamscan
+```
+
+## The nice/ionice Flags
+
+Without throttling, `clamscan -r /` will peg a CPU core for 30–90 minutes depending on disk size and file count. On production hosts this causes Netdata alerts and visible service degradation.
+
+| Flag | Value | Meaning |
+|------|-------|---------|
+| `nice -n 19` | Lowest CPU priority | Kernel will preempt this process for anything else |
+| `ionice -c 3` | Idle I/O class | Disk I/O only runs when no other process needs the disk |
+
+With both flags set, `clamscan` becomes essentially invisible under normal load. The scan takes longer (possibly 2–4× on busy disks), but this is acceptable for a weekly background job.
+
+> **SELinux on Fedora/Fedora:** `ionice` may trigger AVC denials under SELinux Enforcing. If scans silently fail on Fedora hosts, check `ausearch -m avc -ts recent` for `clamscan` denials. See [selinux-fail2ban-execmem-fix](../../05-troubleshooting/selinux-fail2ban-execmem-fix.md) for the pattern.
+
+## Excluded Paths
+
+Always exclude virtual/pseudo filesystems — scanning them wastes time and can trigger false positives or kernel errors:
+
+```
+--exclude-dir=^/proc   # Process info (not real files)
+--exclude-dir=^/sys    # Kernel interfaces
+--exclude-dir=^/dev    # Device nodes
+--exclude-dir=^/run    # Runtime tmpfs
+```
+
+You may also want to exclude large data directories (`/var/lib/docker`, backup volumes, media stores) if scan time is a concern. These are lower-risk targets anyway.
+
+## Quarantine vs Delete
+
+`--move=/var/lib/clamav/quarantine` moves detected files rather than deleting them. This is safer than `--remove` — you can inspect and restore false positives. Review the quarantine directory periodically:
+
+```bash
+ls -la /var/lib/clamav/quarantine/
+```
+
+If a file is a confirmed false positive, restore it and add it to `/etc/clamav/whitelist.ign2`.
+
+## Checking Scan Results
+
+```bash
+# View last scan log
+cat /var/log/clamav/scan.log
+
+# Summary line from the log
+grep -E "^Infected|^Scanned" /var/log/clamav/scan.log | tail -5
+
+# Check freshclam is keeping definitions current
+systemctl status clamav-freshclam
+freshclam --version
+```
+
+## Verifying Deployment
+
+Test that ClamAV can detect malware using the EICAR test file (a harmless string that all AV tools recognize as test malware):
+
+```bash
+echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' \
+  > /tmp/eicar-test.txt
+clamscan /tmp/eicar-test.txt
+# Expected: /tmp/eicar-test.txt: Eicar-Signature FOUND
+rm /tmp/eicar-test.txt
+```
+
+## See Also
+
+- [clamscan-cpu-spike-nice-ionice](../../05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — troubleshooting CPU spikes from unthrottled scans
+- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)
+- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md)

02-selfhosting/security/firewalld-fleet-hardening.md

@@ -0,0 +1,173 @@
+---
+title: Firewall Hardening with firewalld on Fedora Fleet
+domain: selfhosting
+category: security
+tags:
+  - firewall
+  - firewalld
+  - iptables
+  - fedora
+  - ansible
+  - security
+  - hardening
+status: published
+created: 2026-04-18
+updated: 2026-04-18T11:13
+---
+# Firewall Hardening with firewalld on Fedora Fleet
+
+## Overview
+
+Fedora and RHEL-based hosts use `firewalld` as the default firewall manager, backed by `nftables`. Over time, firewall rules accumulate stale entries — decommissioned services, old IP allowances, leftover port forwards — that widen the attack surface silently. This article covers the audit-and-harden pattern for Fedora fleet hosts using Ansible.
+
+> For Ubuntu/Debian hosts, see [ufw-firewall-management](ufw-firewall-management.md).
+
+## The Problem with Accumulated Rules
+
+Rules added manually or by service installers (`firewall-cmd --add-port=...`) don't get cleaned up when services are removed. Common sources of stale rules:
+
+- Monitoring agents (Zabbix, old Netdata exporters)
+- Media servers moved to another host (Jellyfin, Plex)
+- Development ports left open during testing
+- IP-specific allowances for home IPs that have since changed
+
+These stale rules are invisible in day-to-day operation but show up during audits as unnecessary exposure.
+
+## Auditing Current Rules
+
+```bash
+# Show all active rules (nftables, what firewalld actually uses)
+nft list ruleset
+
+# Show firewalld zones and services
+firewall-cmd --list-all-zones
+
+# Show permanent config (what survives reboot)
+firewall-cmd --permanent --list-all
+```
+
+Cross-reference open ports against running services:
+
+```bash
+# What's actually listening?
+ss -tlnp
+
+# Match against firewall rules — anything open that has no listener is stale
+```
+
+## Ansible Hardening Approach
+
+Rather than patching rules incrementally, the cleanest approach is to **flush and rebuild**: remove all non-essential rules and explicitly whitelist only what the host legitimately serves. This avoids drift and makes the resulting ruleset self-documenting.
+
+The Ansible playbook uses `ansible.posix.firewalld` to manage rules declaratively and a flush task to clear the slate before applying the desired state.
+
+### Pattern: Flush → Rebuild
+
+```yaml
+- name: Remove stale firewalld rules
+  ansible.posix.firewalld:
+    port: "{{ item }}"
+    permanent: true
+    state: disabled
+  loop:
+    - 8096/tcp   # Jellyfin — decommissioned
+    - 10050/tcp  # Zabbix agent — removed
+    - 10051/tcp  # Zabbix server — removed
+  ignore_errors: true  # OK if rule doesn't exist
+
+- name: Apply minimal whitelist
+  ansible.posix.firewalld:
+    port: "{{ item }}"
+    permanent: true
+    state: enabled
+  loop: "{{ allowed_ports }}"
+  notify: Reload firewalld
+```
+
+Define `allowed_ports` per host in `host_vars/`:
+
+```yaml
+# host_vars/majorlab/firewall.yml
+allowed_ports:
+  - 80/tcp    # Caddy HTTP
+  - 443/tcp   # Caddy HTTPS
+  - 22/tcp    # SSH (public)
+  - 2222/tcp  # SSH (alt)
+  - 3478/tcp  # Nextcloud Talk TURN
+```
+
+### Tailscale SSH: Restrict to ts-input Zone
+
+For hosts where SSH should only be accessible via Tailscale, move the SSH rule from the public zone to the `ts-input` interface:
+
+```yaml
+- name: Remove SSH from public zone
+  ansible.posix.firewalld:
+    zone: public
+    service: ssh
+    permanent: true
+    state: disabled
+
+- name: Allow SSH on Tailscale interface only
+  ansible.posix.firewalld:
+    zone: trusted
+    interface: tailscale0
+    permanent: true
+    state: enabled
+  notify: Reload firewalld
+```
+
+> **Note:** The Tailscale interface is `tailscale0` unless customized. Confirm with `ip link show` before applying.
+
+## Per-Host Hardening Reference
+
+Different host roles need different rule sets. These are the minimal whitelists for common MajorsHouse host types:
+
+| Host Role | Open Ports | Notes |
+|-----------|-----------|-------|
+| Reverse proxy (Caddy) | 80, 443, 22/2222 | No app ports exposed — Caddy proxies internally |
+| Storage/media (Plex) | 32400 (public), 22 (Tailscale-only) | Plex needs public; SSH Tailscale-only |
+| Bot/Discord host | 25 (Postfix), 25000 (webUI), 6514 (syslog-TLS) | No inbound SSH needed if Tailscale-only |
+| Mail server | 25, 587, 993, 443, 22 | Standard mail ports |
+
+## Default Policy
+
+Set the default zone policy to `DROP` (not `REJECT`) to make the host non-discoverable:
+
+```bash
+firewall-cmd --set-default-zone=drop --permanent
+firewall-cmd --reload
+```
+
+`DROP` silently discards packets; `REJECT` sends an ICMP unreachable back, confirming the host exists.
+
+## Verifying After Apply
+
+```bash
+# Confirm active rules match intent
+firewall-cmd --list-all
+
+# Spot-check a port that should be closed
+nmap -p 10050 <host-ip>
+# Expected: filtered (not open, not closed)
+
+# Confirm a port that should be open
+nmap -p 443 <host-ip>
+# Expected: open
+```
+
+## Ansible Handler
+
+```yaml
+handlers:
+  - name: Reload firewalld
+    ansible.builtin.service:
+      name: firewalld
+      state: reloaded
+```
+
+## See Also
+
+- [ufw-firewall-management](ufw-firewall-management.md) — Ubuntu/Debian equivalent
+- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md)
+- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)

02-selfhosting/services/ghost-smtp-mailgun-setup.md

@@ -0,0 +1,121 @@
+---
+title: Ghost Email Configuration with Mailgun
+domain: selfhosting
+category: services
+tags:
+  - ghost
+  - mailgun
+  - smtp
+  - email
+  - docker
+  - newsletter
+status: published
+created: 2026-04-18
+updated: 2026-04-18T11:13
+---
+# Ghost Email Configuration with Mailgun
+
+## Overview
+
+Ghost uses **two separate mail systems** that must be configured independently. This is the most common source of confusion in Ghost email setup — configuring one does not configure the other.
+
+| System | Purpose | Where configured |
+|--------|---------|-----------------|
+| **Newsletter / Member email** | Sending posts to subscribers | Ghost Admin UI → Settings → Email (stored in DB) |
+| **Transactional / Staff email** | Magic links, password resets, admin notifications | `docker-compose.yml` environment variables |
+
+Both should route through Mailgun for consistent deliverability and tracking.
+
+## Prerequisites
+
+- A Mailgun account with a verified sending domain
+- DNS access for your sending domain
+- Ghost running in Docker (this guide assumes Docker Compose)
+
+## Step 1 — DNS Records
+
+Add these records to your sending domain before configuring Ghost. Mailgun will verify them before allowing sends.
+
+| Type | Name | Value |
+|------|------|-------|
+| TXT | `@` | `v=spf1 include:mailgun.org ~all` |
+| TXT | `pdk1._domainkey` | *(provided by Mailgun — long DKIM key)* |
+| CNAME | `email` | `mailgun.org` |
+
+The tracking CNAME (`email.yourdomain.com`) enables Mailgun's open/click tracking. Ghost's EmailAnalytics feature requires it.
+
+After adding records, verify in Mailgun → Sending → Domains → your domain → DNS Records. All records should show green.
+
+## Step 2 — Newsletter Email (Mailgun API)
+
+Configure in **Ghost Admin → Settings → Email newsletter**. Ghost stores these settings in its database `settings` table — not in the compose file.
+
+| Setting | Value |
+|---------|-------|
+| Mailgun region | US (api.mailgun.net) or EU (api.eu.mailgun.net) |
+| Mailgun domain | `yourdomain.com` |
+| Mailgun API key | Private API key from Mailgun dashboard |
+
+Ghost uses the Mailgun API (not SMTP) for newsletter delivery. This enables open tracking, click tracking, and the EmailAnalytics dashboard.
+
+> **Verify via DB:** If Ghost is MySQL-backed, you can confirm the settings landed:
+> ```bash
+> docker exec <db-container> mysql -u root -p<password> ghost \
+>   -e "SELECT key_name, value FROM settings WHERE key_name LIKE 'mailgun%';"
+> ```
+
+## Step 3 — Transactional Email (SMTP via Mailgun)
+
+Configure in `docker-compose.yml` as environment variables. Ghost's default transport (`Direct`) attempts raw SMTP delivery, which is blocked by most hosting providers and treated as spam. Mailgun SMTP is the reliable path.
+
+```yaml
+services:
+  ghost:
+    image: ghost:6-alpine
+    environment:
+      # ... other Ghost config ...
+      mail__transport: SMTP
+      mail__from: noreply@yourdomain.com
+      mail__options__host: smtp.mailgun.org
+      mail__options__port: 587
+      mail__options__auth__user: postmaster@yourdomain.com
+      mail__options__auth__pass: <mailgun-smtp-password>
+```
+
+The SMTP password is separate from the API key. Find it in Mailgun → Sending → Domains → your domain → SMTP credentials → `postmaster@yourdomain.com`.
+
+After updating the compose file, restart Ghost:
+
+```bash
+cd /root/<stack-dir> && docker compose up -d
+```
+
+Check logs for a clean boot with no mail-related warnings:
+
+```bash
+docker logs <ghost-container> 2>&1 | grep -i mail
+```
+
+## Verifying the Full Stack
+
+**Newsletter:** Send a test post to members (even with 1 subscriber). Check Ghost Admin → Posts → sent post → Email analytics. Delivered count should increment within minutes.
+
+**Transactional:** Trigger a staff magic link (Ghost Admin → sign out → request magic link). The email should arrive within seconds.
+
+**Mailgun logs:** Mailgun → Logs → Events shows all API and SMTP activity. Filter by domain to isolate Ghost sends.
+
+## Common Issues
+
+**Newsletter sends but staff emails don't arrive (or vice versa):** The two systems are independent. Check both configurations separately.
+
+**`transport: Direct` in config:** Ghost writes a `config.production.json` inside the container. If `mail.transport` shows `Direct`, the environment variables didn't apply — verify the compose key names (double underscores for nested config).
+
+**Mailgun API key vs SMTP password:** These are different credentials. The API key (starts with `key-`) is for the newsletter system. The SMTP password is for the transactional system. Don't mix them.
+
+**Domain state: `unverified` in Mailgun:** DNS records haven't propagated or are wrong. Use `dig TXT yourdomain.com` and `dig TXT pdk1._domainkey.yourdomain.com` to verify from outside your network.
+
+## See Also
+
+- [ghost-emailanalytics-lag-warning](../../05-troubleshooting/ghost-emailanalytics-lag-warning.md)
+- [docker-healthchecks](../docker/docker-healthchecks.md)
+- [watchtower-smtp-localhost-relay](../docker/watchtower-smtp-localhost-relay.md)