diff --git a/.gitattributes b/.gitattributes deleted file mode 100644 index df71be0..0000000 --- a/.gitattributes +++ /dev/null @@ -1,18 +0,0 @@ -# Normalize line endings to LF for all text files -* text=auto eol=lf - -# Explicitly handle markdown -*.md text eol=lf - -# Explicitly handle config files -*.yml text eol=lf -*.yaml text eol=lf -*.json text eol=lf -*.toml text eol=lf - -# Binary files — don't touch -*.png binary -*.jpg binary -*.jpeg binary -*.gif binary -*.pdf binary diff --git a/.gitignore b/.gitignore deleted file mode 100644 index ba858bb..0000000 --- a/.gitignore +++ /dev/null @@ -1,8 +0,0 @@ -# Obsidian specific -.obsidian/workspace.json -.obsidian/workspace-mobile.json -.obsidian/cache/ - -# Windows/WSL specific -Thumbs.db -.DS_Store diff --git a/0 b/0 deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/distro-specific/.keep b/01-linux/distro-specific/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/files-permissions/.keep b/01-linux/files-permissions/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/networking/.keep b/01-linux/networking/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/networking/ssh-config-key-management.md b/01-linux/networking/ssh-config-key-management.md index 5962611..37ef3fc 100644 --- a/01-linux/networking/ssh-config-key-management.md +++ b/01-linux/networking/ssh-config-key-management.md @@ -10,7 +10,7 @@ tags: - remote-access status: published created: 2026-03-08 -updated: 2026-04-07T21:55 +updated: 2026-04-14T14:27 --- # SSH Config and Key Management diff --git a/01-linux/packages/.keep b/01-linux/packages/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/process-management/.keep b/01-linux/process-management/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/shell-scripting/.keep b/01-linux/shell-scripting/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/01-linux/storage/.keep b/01-linux/storage/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/dns-networking/.keep b/02-selfhosting/dns-networking/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/docker/.keep b/02-selfhosting/docker/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/monitoring/.keep b/02-selfhosting/monitoring/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/reverse-proxy/.keep b/02-selfhosting/reverse-proxy/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/security/.keep b/02-selfhosting/security/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/security/clamav-fleet-deployment.md b/02-selfhosting/security/clamav-fleet-deployment.md new file mode 100644 index 0000000..373c75f --- /dev/null +++ b/02-selfhosting/security/clamav-fleet-deployment.md @@ -0,0 +1,154 @@ +--- +title: ClamAV Fleet Deployment with Ansible +domain: selfhosting +category: security +tags: + - clamav + - antivirus + - security + - ansible + - fleet + - cron +status: published +created: 2026-04-18 +updated: 2026-04-18T11:13 +--- +# ClamAV Fleet Deployment with Ansible + +## Overview + +ClamAV is the standard open-source antivirus for Linux servers. For internet-facing hosts, a weekly scan with fresh definitions catches known malware, web shells, and suspicious files before they cause damage. The key operational concern is CPU impact — an unthrottled `clamscan` will saturate a core for hours on a busy host. The solution is `nice` and `ionice` wrappers. + +> This guide covers deployment to internet-facing hosts. Internal-only hosts (storage, inference, gaming) are lower priority and can be skipped. + +## What Gets Deployed + +- `clamav` + `clamav-update` packages (provides `clamscan` + `freshclam`) +- `freshclam` service enabled for automatic definition updates +- A quarantine directory at `/var/lib/clamav/quarantine/` +- A weekly `clamscan` cron job, niced to background priority +- SELinux context set on the quarantine directory (Fedora hosts) + +## Ansible Playbook + +```yaml +- name: Deploy ClamAV to internet-facing hosts + hosts: internet_facing # dca, majorlinux, teelia, tttpod, majortoot, majormail + become: true + + tasks: + + - name: Install ClamAV packages + ansible.builtin.package: + name: + - clamav + - clamav-update + state: present + + - name: Enable and start freshclam + ansible.builtin.service: + name: clamav-freshclam + enabled: true + state: started + + - name: Create quarantine directory + ansible.builtin.file: + path: /var/lib/clamav/quarantine + state: directory + owner: root + group: root + mode: '0700' + + - name: Set SELinux context on quarantine dir (Fedora/RHEL) + ansible.builtin.command: + cmd: chcon -t var_t /var/lib/clamav/quarantine + when: ansible_os_family == "RedHat" + changed_when: false + + - name: Deploy weekly clamscan cron job + ansible.builtin.cron: + name: "Weekly ClamAV scan" + user: root + weekday: "0" # Sunday + hour: "3" + minute: "0" + job: >- + nice -n 19 ionice -c 3 + clamscan -r / + --exclude-dir=^/proc + --exclude-dir=^/sys + --exclude-dir=^/dev + --exclude-dir=^/run + --move=/var/lib/clamav/quarantine + --log=/var/log/clamav/scan.log + --quiet + 2>&1 | logger -t clamscan +``` + +## The nice/ionice Flags + +Without throttling, `clamscan -r /` will peg a CPU core for 30–90 minutes depending on disk size and file count. On production hosts this causes Netdata alerts and visible service degradation. + +| Flag | Value | Meaning | +|------|-------|---------| +| `nice -n 19` | Lowest CPU priority | Kernel will preempt this process for anything else | +| `ionice -c 3` | Idle I/O class | Disk I/O only runs when no other process needs the disk | + +With both flags set, `clamscan` becomes essentially invisible under normal load. The scan takes longer (possibly 2–4× on busy disks), but this is acceptable for a weekly background job. + +> **SELinux on Fedora/Fedora:** `ionice` may trigger AVC denials under SELinux Enforcing. If scans silently fail on Fedora hosts, check `ausearch -m avc -ts recent` for `clamscan` denials. See [selinux-fail2ban-execmem-fix](../../05-troubleshooting/selinux-fail2ban-execmem-fix.md) for the pattern. + +## Excluded Paths + +Always exclude virtual/pseudo filesystems — scanning them wastes time and can trigger false positives or kernel errors: + +``` +--exclude-dir=^/proc # Process info (not real files) +--exclude-dir=^/sys # Kernel interfaces +--exclude-dir=^/dev # Device nodes +--exclude-dir=^/run # Runtime tmpfs +``` + +You may also want to exclude large data directories (`/var/lib/docker`, backup volumes, media stores) if scan time is a concern. These are lower-risk targets anyway. + +## Quarantine vs Delete + +`--move=/var/lib/clamav/quarantine` moves detected files rather than deleting them. This is safer than `--remove` — you can inspect and restore false positives. Review the quarantine directory periodically: + +```bash +ls -la /var/lib/clamav/quarantine/ +``` + +If a file is a confirmed false positive, restore it and add it to `/etc/clamav/whitelist.ign2`. + +## Checking Scan Results + +```bash +# View last scan log +cat /var/log/clamav/scan.log + +# Summary line from the log +grep -E "^Infected|^Scanned" /var/log/clamav/scan.log | tail -5 + +# Check freshclam is keeping definitions current +systemctl status clamav-freshclam +freshclam --version +``` + +## Verifying Deployment + +Test that ClamAV can detect malware using the EICAR test file (a harmless string that all AV tools recognize as test malware): + +```bash +echo 'X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*' \ + > /tmp/eicar-test.txt +clamscan /tmp/eicar-test.txt +# Expected: /tmp/eicar-test.txt: Eicar-Signature FOUND +rm /tmp/eicar-test.txt +``` + +## See Also + +- [clamscan-cpu-spike-nice-ionice](../../05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — troubleshooting CPU spikes from unthrottled scans +- [linux-server-hardening-checklist](linux-server-hardening-checklist.md) +- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md) diff --git a/02-selfhosting/security/firewalld-fleet-hardening.md b/02-selfhosting/security/firewalld-fleet-hardening.md new file mode 100644 index 0000000..4f77186 --- /dev/null +++ b/02-selfhosting/security/firewalld-fleet-hardening.md @@ -0,0 +1,173 @@ +--- +title: Firewall Hardening with firewalld on Fedora Fleet +domain: selfhosting +category: security +tags: + - firewall + - firewalld + - iptables + - fedora + - ansible + - security + - hardening +status: published +created: 2026-04-18 +updated: 2026-04-18T11:13 +--- +# Firewall Hardening with firewalld on Fedora Fleet + +## Overview + +Fedora and RHEL-based hosts use `firewalld` as the default firewall manager, backed by `nftables`. Over time, firewall rules accumulate stale entries — decommissioned services, old IP allowances, leftover port forwards — that widen the attack surface silently. This article covers the audit-and-harden pattern for Fedora fleet hosts using Ansible. + +> For Ubuntu/Debian hosts, see [ufw-firewall-management](ufw-firewall-management.md). + +## The Problem with Accumulated Rules + +Rules added manually or by service installers (`firewall-cmd --add-port=...`) don't get cleaned up when services are removed. Common sources of stale rules: + +- Monitoring agents (Zabbix, old Netdata exporters) +- Media servers moved to another host (Jellyfin, Plex) +- Development ports left open during testing +- IP-specific allowances for home IPs that have since changed + +These stale rules are invisible in day-to-day operation but show up during audits as unnecessary exposure. + +## Auditing Current Rules + +```bash +# Show all active rules (nftables, what firewalld actually uses) +nft list ruleset + +# Show firewalld zones and services +firewall-cmd --list-all-zones + +# Show permanent config (what survives reboot) +firewall-cmd --permanent --list-all +``` + +Cross-reference open ports against running services: + +```bash +# What's actually listening? +ss -tlnp + +# Match against firewall rules — anything open that has no listener is stale +``` + +## Ansible Hardening Approach + +Rather than patching rules incrementally, the cleanest approach is to **flush and rebuild**: remove all non-essential rules and explicitly whitelist only what the host legitimately serves. This avoids drift and makes the resulting ruleset self-documenting. + +The Ansible playbook uses `ansible.posix.firewalld` to manage rules declaratively and a flush task to clear the slate before applying the desired state. + +### Pattern: Flush → Rebuild + +```yaml +- name: Remove stale firewalld rules + ansible.posix.firewalld: + port: "{{ item }}" + permanent: true + state: disabled + loop: + - 8096/tcp # Jellyfin — decommissioned + - 10050/tcp # Zabbix agent — removed + - 10051/tcp # Zabbix server — removed + ignore_errors: true # OK if rule doesn't exist + +- name: Apply minimal whitelist + ansible.posix.firewalld: + port: "{{ item }}" + permanent: true + state: enabled + loop: "{{ allowed_ports }}" + notify: Reload firewalld +``` + +Define `allowed_ports` per host in `host_vars/`: + +```yaml +# host_vars/majorlab/firewall.yml +allowed_ports: + - 80/tcp # Caddy HTTP + - 443/tcp # Caddy HTTPS + - 22/tcp # SSH (public) + - 2222/tcp # SSH (alt) + - 3478/tcp # Nextcloud Talk TURN +``` + +### Tailscale SSH: Restrict to ts-input Zone + +For hosts where SSH should only be accessible via Tailscale, move the SSH rule from the public zone to the `ts-input` interface: + +```yaml +- name: Remove SSH from public zone + ansible.posix.firewalld: + zone: public + service: ssh + permanent: true + state: disabled + +- name: Allow SSH on Tailscale interface only + ansible.posix.firewalld: + zone: trusted + interface: tailscale0 + permanent: true + state: enabled + notify: Reload firewalld +``` + +> **Note:** The Tailscale interface is `tailscale0` unless customized. Confirm with `ip link show` before applying. + +## Per-Host Hardening Reference + +Different host roles need different rule sets. These are the minimal whitelists for common MajorsHouse host types: + +| Host Role | Open Ports | Notes | +|-----------|-----------|-------| +| Reverse proxy (Caddy) | 80, 443, 22/2222 | No app ports exposed — Caddy proxies internally | +| Storage/media (Plex) | 32400 (public), 22 (Tailscale-only) | Plex needs public; SSH Tailscale-only | +| Bot/Discord host | 25 (Postfix), 25000 (webUI), 6514 (syslog-TLS) | No inbound SSH needed if Tailscale-only | +| Mail server | 25, 587, 993, 443, 22 | Standard mail ports | + +## Default Policy + +Set the default zone policy to `DROP` (not `REJECT`) to make the host non-discoverable: + +```bash +firewall-cmd --set-default-zone=drop --permanent +firewall-cmd --reload +``` + +`DROP` silently discards packets; `REJECT` sends an ICMP unreachable back, confirming the host exists. + +## Verifying After Apply + +```bash +# Confirm active rules match intent +firewall-cmd --list-all + +# Spot-check a port that should be closed +nmap -p 10050 +# Expected: filtered (not open, not closed) + +# Confirm a port that should be open +nmap -p 443 +# Expected: open +``` + +## Ansible Handler + +```yaml +handlers: + - name: Reload firewalld + ansible.builtin.service: + name: firewalld + state: reloaded +``` + +## See Also + +- [ufw-firewall-management](ufw-firewall-management.md) — Ubuntu/Debian equivalent +- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md) +- [linux-server-hardening-checklist](linux-server-hardening-checklist.md) diff --git a/02-selfhosting/services/.keep b/02-selfhosting/services/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/02-selfhosting/services/ghost-smtp-mailgun-setup.md b/02-selfhosting/services/ghost-smtp-mailgun-setup.md new file mode 100644 index 0000000..822cf7a --- /dev/null +++ b/02-selfhosting/services/ghost-smtp-mailgun-setup.md @@ -0,0 +1,121 @@ +--- +title: Ghost Email Configuration with Mailgun +domain: selfhosting +category: services +tags: + - ghost + - mailgun + - smtp + - email + - docker + - newsletter +status: published +created: 2026-04-18 +updated: 2026-04-18T11:13 +--- +# Ghost Email Configuration with Mailgun + +## Overview + +Ghost uses **two separate mail systems** that must be configured independently. This is the most common source of confusion in Ghost email setup — configuring one does not configure the other. + +| System | Purpose | Where configured | +|--------|---------|-----------------| +| **Newsletter / Member email** | Sending posts to subscribers | Ghost Admin UI → Settings → Email (stored in DB) | +| **Transactional / Staff email** | Magic links, password resets, admin notifications | `docker-compose.yml` environment variables | + +Both should route through Mailgun for consistent deliverability and tracking. + +## Prerequisites + +- A Mailgun account with a verified sending domain +- DNS access for your sending domain +- Ghost running in Docker (this guide assumes Docker Compose) + +## Step 1 — DNS Records + +Add these records to your sending domain before configuring Ghost. Mailgun will verify them before allowing sends. + +| Type | Name | Value | +|------|------|-------| +| TXT | `@` | `v=spf1 include:mailgun.org ~all` | +| TXT | `pdk1._domainkey` | *(provided by Mailgun — long DKIM key)* | +| CNAME | `email` | `mailgun.org` | + +The tracking CNAME (`email.yourdomain.com`) enables Mailgun's open/click tracking. Ghost's EmailAnalytics feature requires it. + +After adding records, verify in Mailgun → Sending → Domains → your domain → DNS Records. All records should show green. + +## Step 2 — Newsletter Email (Mailgun API) + +Configure in **Ghost Admin → Settings → Email newsletter**. Ghost stores these settings in its database `settings` table — not in the compose file. + +| Setting | Value | +|---------|-------| +| Mailgun region | US (api.mailgun.net) or EU (api.eu.mailgun.net) | +| Mailgun domain | `yourdomain.com` | +| Mailgun API key | Private API key from Mailgun dashboard | + +Ghost uses the Mailgun API (not SMTP) for newsletter delivery. This enables open tracking, click tracking, and the EmailAnalytics dashboard. + +> **Verify via DB:** If Ghost is MySQL-backed, you can confirm the settings landed: +> ```bash +> docker exec mysql -u root -p ghost \ +> -e "SELECT key_name, value FROM settings WHERE key_name LIKE 'mailgun%';" +> ``` + +## Step 3 — Transactional Email (SMTP via Mailgun) + +Configure in `docker-compose.yml` as environment variables. Ghost's default transport (`Direct`) attempts raw SMTP delivery, which is blocked by most hosting providers and treated as spam. Mailgun SMTP is the reliable path. + +```yaml +services: + ghost: + image: ghost:6-alpine + environment: + # ... other Ghost config ... + mail__transport: SMTP + mail__from: noreply@yourdomain.com + mail__options__host: smtp.mailgun.org + mail__options__port: 587 + mail__options__auth__user: postmaster@yourdomain.com + mail__options__auth__pass: +``` + +The SMTP password is separate from the API key. Find it in Mailgun → Sending → Domains → your domain → SMTP credentials → `postmaster@yourdomain.com`. + +After updating the compose file, restart Ghost: + +```bash +cd /root/ && docker compose up -d +``` + +Check logs for a clean boot with no mail-related warnings: + +```bash +docker logs 2>&1 | grep -i mail +``` + +## Verifying the Full Stack + +**Newsletter:** Send a test post to members (even with 1 subscriber). Check Ghost Admin → Posts → sent post → Email analytics. Delivered count should increment within minutes. + +**Transactional:** Trigger a staff magic link (Ghost Admin → sign out → request magic link). The email should arrive within seconds. + +**Mailgun logs:** Mailgun → Logs → Events shows all API and SMTP activity. Filter by domain to isolate Ghost sends. + +## Common Issues + +**Newsletter sends but staff emails don't arrive (or vice versa):** The two systems are independent. Check both configurations separately. + +**`transport: Direct` in config:** Ghost writes a `config.production.json` inside the container. If `mail.transport` shows `Direct`, the environment variables didn't apply — verify the compose key names (double underscores for nested config). + +**Mailgun API key vs SMTP password:** These are different credentials. The API key (starts with `key-`) is for the newsletter system. The SMTP password is for the transactional system. Don't mix them. + +**Domain state: `unverified` in Mailgun:** DNS records haven't propagated or are wrong. Use `dig TXT yourdomain.com` and `dig TXT pdk1._domainkey.yourdomain.com` to verify from outside your network. + +## See Also + +- [ghost-emailanalytics-lag-warning](../../05-troubleshooting/ghost-emailanalytics-lag-warning.md) +- [docker-healthchecks](../docker/docker-healthchecks.md) +- [watchtower-smtp-localhost-relay](../docker/watchtower-smtp-localhost-relay.md) diff --git a/02-selfhosting/storage-backup/.keep b/02-selfhosting/storage-backup/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/03-opensource/alternatives/.keep b/03-opensource/alternatives/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/03-opensource/dev-tools/.keep b/03-opensource/dev-tools/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/03-opensource/media-creative/.keep b/03-opensource/media-creative/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/03-opensource/privacy-security/.keep b/03-opensource/privacy-security/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/03-opensource/productivity/.keep b/03-opensource/productivity/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/04-streaming/audio/.keep b/04-streaming/audio/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/04-streaming/hardware/.keep b/04-streaming/hardware/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/04-streaming/infrastructure/.keep b/04-streaming/infrastructure/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/04-streaming/obs/.keep b/04-streaming/obs/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/04-streaming/podcast/.keep b/04-streaming/podcast/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/ansible-check-mode-false-positives.md b/05-troubleshooting/ansible-check-mode-false-positives.md new file mode 100644 index 0000000..7831188 --- /dev/null +++ b/05-troubleshooting/ansible-check-mode-false-positives.md @@ -0,0 +1,135 @@ +--- +title: Ansible Check Mode False Positives in Verify/Assert Tasks +domain: selfhosting +category: troubleshooting +tags: + - ansible + - check-mode + - dry-run + - assert + - handlers + - troubleshooting +status: published +created: 2026-04-18 +updated: 2026-04-18T11:13 +--- +# Ansible Check Mode False Positives in Verify/Assert Tasks + +## The Problem + +`ansible-playbook --check` (dry-run mode) reports failures on verify and assert tasks that depend on handler-triggered side effects — even when the playbook is correct and would succeed on a real run. + +**Symptom:** Running `--check` produces errors like: + +``` +TASK [Assert hardened settings are active] *** +fatal: [host]: FAILED! => { + "assertion": "'permitrootlogin without-password' in sshd_effective.stdout", + "msg": "One or more SSH hardening settings not effective" +} +``` + +But a real run (`ansible-playbook` without `--check`) succeeds cleanly. + +## Why It Happens + +In check mode, Ansible simulates tasks but **does not execute handlers**. This means: + +1. A config file task reports `changed` (it would deploy the file) +2. The handler (`reload sshd`, `reload firewalld`, etc.) is **never fired** +3. A subsequent verify task runs `sshd -T` or `ufw status verbose` against the **current live state** (pre-change) +4. The assert compares the current state against the expected post-change state and fails + +The verify task is reading reality accurately — the change hasn't happened yet — but the failure is misleading. It suggests the playbook is broken when it's actually correct. + +## The Fix + +Guard verify and assert tasks that depend on handler side effects with `when: not ansible_check_mode`: + +```yaml +- name: Verify effective SSH settings post-reload + ansible.builtin.command: + cmd: sshd -T + register: sshd_effective + changed_when: false + when: not ansible_check_mode # sshd hasn't reloaded in check mode + +- name: Assert hardened settings are active + ansible.builtin.assert: + that: + - "'permitrootlogin without-password' in sshd_effective.stdout" + - "'x11forwarding no' in sshd_effective.stdout" + fail_msg: "SSH hardening settings not effective — check for conflicting config" + when: not ansible_check_mode # result would be pre-change state +``` + +This skips the verify/assert during check mode (where they'd produce false failures) while keeping them active on real runs (where they catch actual misconfigurations). + +## When to Apply This Guard + +Apply `when: not ansible_check_mode` to any task that: + +- Reads the **active/effective state** of a service after a config change (`sshd -T`, `ufw status verbose`, `firewall-cmd --list-all`, `nginx -T`) +- **Asserts** that the post-change state matches expectations +- Depends on a **handler** having fired first (service reload, daemon restart) + +Don't apply it to tasks that check pre-existing state (e.g., verifying a file exists before modifying it) — those are valid in check mode. + +## Common Patterns + +### SSH daemon verify + +```yaml +- name: Verify effective sshd settings + ansible.builtin.command: sshd -T + register: sshd_out + changed_when: false + when: not ansible_check_mode + +- name: Assert sshd hardening active + ansible.builtin.assert: + that: + - "'maxauthtries 3' in sshd_out.stdout" + when: not ansible_check_mode +``` + +### UFW status verify + +```yaml +- name: Show UFW status + ansible.builtin.command: ufw status verbose + register: ufw_status + changed_when: false + when: not ansible_check_mode + +- name: Confirm default deny incoming + ansible.builtin.assert: + that: + - "'Default: deny (incoming)' in ufw_status.stdout" + when: not ansible_check_mode +``` + +### nginx config verify + +```yaml +- name: Test nginx config + ansible.builtin.command: nginx -t + changed_when: false + when: not ansible_check_mode +``` + +## Trade-off + +Guarding with `when: not ansible_check_mode` means check mode won't validate these assertions. The benefit — no false failures — outweighs the gap because: + +- Check mode is showing you what *would* change, not whether the result is valid +- Real runs still assert and will catch actual misconfigurations +- The alternative (failing check runs) erodes trust in `--check` output + +If you need to verify the effective post-change state in check mode, consider splitting the playbook into a deploy pass and a separate verify-only playbook run without `--check`. + +## See Also + +- [ssh-hardening-ansible-fleet](../02-selfhosting/security/ssh-hardening-ansible-fleet.md) +- [ufw-firewall-management](../02-selfhosting/security/ufw-firewall-management.md) +- [ansible-getting-started](../01-linux/shell-scripting/ansible-getting-started.md) diff --git a/05-troubleshooting/boot-system/.keep b/05-troubleshooting/boot-system/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/docker/.keep b/05-troubleshooting/docker/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/ghost-emailanalytics-lag-warning.md b/05-troubleshooting/ghost-emailanalytics-lag-warning.md new file mode 100644 index 0000000..bfcc9bd --- /dev/null +++ b/05-troubleshooting/ghost-emailanalytics-lag-warning.md @@ -0,0 +1,106 @@ +--- +title: Ghost EmailAnalytics Lag Warning — What It Means and When to Worry +domain: selfhosting +category: troubleshooting +tags: + - ghost + - email + - mailgun + - emailanalytics + - docker + - troubleshooting +status: published +created: 2026-04-18 +updated: 2026-04-18T11:13 +--- +# Ghost EmailAnalytics Lag Warning — What It Means and When to Worry + +## The Warning + +Ghost logs a recurring warning every 5 minutes when its EmailAnalytics job falls behind: + +``` +WARN [EmailAnalytics] Opened events processing is 738.0 minutes behind (threshold: 30) +``` + +This is followed by: + +``` +INFO [EmailAnalytics] Job complete - No events +INFO [EmailAnalytics] Skipping fetchMissing because end (...) is before begin (...) +``` + +The counter increments by 5 with every cycle. On a small newsletter, it will grow indefinitely and never reset on its own — until a subscriber opens an email or a new newsletter is sent. + +## Why It Happens + +Ghost's EmailAnalytics polls Mailgun every 5 minutes for new "opened" events. The cursor is anchored to the timestamp of the last email delivery. If no new opened events arrive from Mailgun, the cursor doesn't advance and the lag counter grows. + +This is **expected behavior** when: +- All subscribers have already opened (their open was recorded) +- One or more subscribers have not opened the email and haven't opened any subsequent emails +- There are no new emails to send + +The lag counter = time since the last opened event was recorded, not time since the last email was sent. + +## The `fetchMissing end == begin` Skip + +``` +INFO [EmailAnalytics] Skipping fetchMissing because end (Fri Apr 17 2026 15:44:57 ...) is before begin (Fri Apr 17 2026 15:44:57 ...) +``` + +This fires when the cursor window collapses to zero width — the start and end of the query window are identical. Ghost's guard clause skips a nonsensical zero-width Mailgun API call. This is not a bug or data loss — it's a safety check. + +## What `status: submitted` Means + +In Ghost's `emails` database table, all successfully sent newsletters show `status: submitted`. This is the normal terminal state after Ghost hands the email batch off to Mailgun. There is no `status: sent` — `submitted` = success. + +You can verify delivery success by checking the counts: + +```bash +docker exec mysql -u root -p ghost \ + -e "SELECT subject, status, email_count, delivered_count, opened_count, failed_count FROM emails ORDER BY created_at DESC LIMIT 5;" +``` + +A healthy result: `email_count == delivered_count`, `failed_count == 0`, regardless of `opened_count`. + +## When to Actually Worry + +The lag warning is **benign** in these cases: +- `delivered_count == email_count` (all emails delivered) +- `failed_count == 0` +- Mailgun domain state is active +- The warning appeared after a successful send and has been growing since + +Investigate further if: +- `delivered_count < email_count` — some emails never left Mailgun +- `failed_count > 0` — delivery failures +- The warning appeared immediately after a Ghost upgrade or Mailgun credential change +- Mailgun Events API shows 0 delivered events (not just 0 opened events) for the send window + +## Checking Mailgun Directly + +If you suspect the lag reflects a real delivery problem, query Mailgun's Events API: + +```bash +# Check for delivered events in the send window +curl -s --user "api:" \ + "https://api.mailgun.net/v3//events?event=delivered&begin=&limit=10" \ + | python3 -m json.tool | grep -E "event|recipient|timestamp" | head -30 +``` + +If delivered events appear for your subscribers, Mailgun is working and the lag warning is purely cosmetic. + +## How It Resolves + +The lag warning self-resolves when: +1. **A subscriber opens an email** — Mailgun returns an "opened" event, the cursor advances, lag resets +2. **A new newsletter is sent** — the send triggers a fresh analytics cycle, cursor jumps forward +3. **Manually resetting the cursor** — possible via direct DB update, but not recommended unless you understand the implications for analytics continuity + +For small newsletters (2–10 subscribers) where one subscriber consistently doesn't open emails, the warning is permanent background noise between sends. It does not indicate data loss or misconfiguration. + +## See Also + +- [ghost-smtp-mailgun-setup](../02-selfhosting/services/ghost-smtp-mailgun-setup.md) +- [debugging-broken-docker-containers](../02-selfhosting/docker/debugging-broken-docker-containers.md) diff --git a/05-troubleshooting/gpu-display/.keep b/05-troubleshooting/gpu-display/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 99c323e..f1470a7 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -1,6 +1,6 @@ --- created: 2026-03-15T06:37 -updated: 2026-04-17T09:57 +updated: 2026-04-17T10:21 --- # 🔧 General Troubleshooting diff --git a/05-troubleshooting/networking/.keep b/05-troubleshooting/networking/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md b/05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md index 4ca9a85..7908bbd 100644 --- a/05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md +++ b/05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md @@ -11,7 +11,7 @@ tags: - powershell status: published created: 2026-04-03 -updated: 2026-04-07T21:55 +updated: 2026-04-14T14:27 --- # Windows OpenSSH: WSL as Default Shell Breaks Remote Commands diff --git a/05-troubleshooting/networking/windows-sshd-stops-after-reboot.md b/05-troubleshooting/networking/windows-sshd-stops-after-reboot.md index 63d449f..801860c 100644 --- a/05-troubleshooting/networking/windows-sshd-stops-after-reboot.md +++ b/05-troubleshooting/networking/windows-sshd-stops-after-reboot.md @@ -10,7 +10,7 @@ tags: - majorrig status: published created: 2026-04-02 -updated: 2026-04-07T21:58 +updated: 2026-04-14T14:27 --- # Windows OpenSSH Server (sshd) Stops After Reboot diff --git a/05-troubleshooting/performance/.keep b/05-troubleshooting/performance/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/05-troubleshooting/yt-dlp-fedora-js-challenge.md b/05-troubleshooting/yt-dlp-fedora-js-challenge.md index a9f89a4..931cb62 100644 --- a/05-troubleshooting/yt-dlp-fedora-js-challenge.md +++ b/05-troubleshooting/yt-dlp-fedora-js-challenge.md @@ -1,11 +1,16 @@ --- -title: "yt-dlp YouTube JS Challenge Fix (Fedora)" +title: yt-dlp YouTube JS Challenge Fix (Fedora) domain: troubleshooting category: general -tags: [yt-dlp, fedora, youtube, javascript, deno] +tags: + - yt-dlp + - fedora + - youtube + - javascript + - deno status: published created: 2026-04-02 -updated: 2026-04-02 +updated: 2026-04-14T14:27 --- # yt-dlp YouTube JS Challenge Fix (Fedora) diff --git a/README.md b/README.md index e59cb59..efa7298 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ --- created: 2026-04-06T09:52 -updated: 2026-04-13T10:16 +updated: 2026-04-14T14:12 --- # MajorLinux Tech Wiki — Index diff --git a/SUMMARY.md b/SUMMARY.md index 5e15279..1f74f3c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -21,7 +21,6 @@ updated: 2026-04-13T10:16 * [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) * [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md) * [Docker Healthchecks](02-selfhosting/docker/docker-healthchecks.md) - * [Watchtower SMTP via Localhost Postfix Relay](02-selfhosting/docker/watchtower-smtp-localhost-relay.md) * [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) * [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) * [Network Overview](02-selfhosting/dns-networking/network-overview.md) @@ -40,9 +39,6 @@ updated: 2026-04-13T10:16 * [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) * [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) * [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) - * [Fail2ban: Enable the nginx-bad-request Jail](02-selfhosting/security/fail2ban-nginx-bad-request-jail.md) - * [Fail2ban Custom Jail: Apache Bad Request Detection](02-selfhosting/security/fail2ban-apache-bad-request-jail.md) - * [SSH Hardening Fleet-Wide with Ansible](02-selfhosting/security/ssh-hardening-ansible-fleet.md) * [Open Source & Alternatives](03-opensource/index.md) * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) diff --git a/index.md b/index.md index fa3207d..4060d4c 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ --- created: 2026-04-06T09:52 -updated: 2026-04-13T10:16 +updated: 2026-04-14T14:12 --- # MajorLinux Tech Wiki — Index