SUMMARY: add 4 new articles to nav (nginx/apache bad-request, SSH hardening, Watchtower relay)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

01-linux/distro-specific/linux-distro-guide-beginners.md

@@ -86,5 +86,5 @@ Be specific when asking for help. Include your distro and version, what you trie
 
 ## See Also
 
-- [[wsl2-instance-migration-fedora43]]
+- [wsl2-instance-migration-fedora43](wsl2-instance-migration-fedora43.md)
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../process-management/managing-linux-services-systemd-ansible.md)

01-linux/distro-specific/wsl2-backup-powershell.md

@@ -72,7 +72,7 @@ wsl --unregister FedoraLinux-43
 wsl --import FedoraLinux-43 D:\WSL\Fedora43 D:\WSL\Backups\FedoraLinux-43-YYYY-MM-DD.tar
 ```
 
-Then fix the default user — after import WSL resets to root. See [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]] for the `/etc/wsl.conf` fix.
+Then fix the default user — after import WSL resets to root. See [WSL2 Instance Migration](wsl2-instance-migration-fedora43.md) for the `/etc/wsl.conf` fix.
 
 ## Gotchas
 
@@ -82,5 +82,5 @@ Then fix the default user — after import WSL resets to root. See [[wsl2-instan
 
 ## See Also
 
-- [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]]
+- [WSL2 Instance Migration](wsl2-instance-migration-fedora43.md)
-- [[wsl2-rebuild-fedora43-training-env|WSL2 Training Environment Rebuild]]
+- [WSL2 Training Environment Rebuild](wsl2-rebuild-fedora43-training-env.md)

01-linux/distro-specific/wsl2-instance-migration-fedora43.md

@@ -97,5 +97,5 @@ alias clean='sudo dnf clean all'
 
 ## See Also
 
-- [[Managing disk space on MajorRig]]
+- Managing disk space on MajorRig
-- [[Unsloth QLoRA fine-tuning setup]]
+- Unsloth QLoRA fine-tuning setup

01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md

@@ -199,5 +199,5 @@ D:\WSL\
 
 ## See Also
 
-- [[wsl2-instance-migration-fedora43|WSL2 Instance Migration]]
+- [WSL2 Instance Migration](wsl2-instance-migration-fedora43.md)
-- [[wsl2-backup-powershell|WSL2 Backup via PowerShell]]
+- [WSL2 Backup via PowerShell](wsl2-backup-powershell.md)

01-linux/files-permissions/linux-file-permissions.md

@@ -152,6 +152,6 @@ find /var/www/html -type d -exec chmod 755 {} \;
 
 ## See Also
 
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../../02-selfhosting/security/linux-server-hardening-checklist.md)
-- [[ssh-config-key-management]]
+- [ssh-config-key-management](../networking/ssh-config-key-management.md)
-- [[bash-scripting-patterns]]
+- [bash-scripting-patterns](../shell-scripting/bash-scripting-patterns.md)

01-linux/networking/ssh-config-key-management.md

@@ -1,11 +1,16 @@
 ---
-title: "SSH Config and Key Management"
+title: SSH Config and Key Management
 domain: linux
 category: networking
-tags: [ssh, keys, security, linux, remote-access]
+tags:
+  - ssh
+  - keys
+  - security
+  - linux
+  - remote-access
 status: published
 created: 2026-03-08
-updated: 2026-03-08
+updated: 2026-04-07T21:55
 ---
 
 # SSH Config and Key Management
@@ -129,7 +134,51 @@ If key auth isn't working and the config looks right, permissions are the first
 - **`ServerAliveInterval` in your config** keeps connections from timing out on idle sessions. Saves you from the annoyance of reconnecting after stepping away.
 - **Never put private keys in cloud storage, Git repos, or Docker images.** It happens more than you'd think.
 
+## Windows OpenSSH: Admin User Key Auth
+
+Windows OpenSSH has a separate key file for users in the `Administrators` group. Regular `~/.ssh/authorized_keys` is **ignored** for admin users unless the `Match Group administrators` block in `sshd_config` is disabled.
+
+### Where keys go
+
+| User type | Key file |
+|---|---|
+| Regular user | `C:\Users\<user>\.ssh\authorized_keys` |
+| Admin user | `C:\ProgramData\ssh\administrators_authorized_keys` |
+
+### Setup (elevated PowerShell)
+
+1. **Enable the Match block** in `C:\ProgramData\ssh\sshd_config` — both lines must be uncommented:
+   ```
+   Match Group administrators
+          AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
+   ```
+
+2. **Write the key file without BOM** — PowerShell 5 defaults to UTF-16LE or UTF-8 with BOM, both of which OpenSSH silently rejects:
+   ```powershell
+   [System.IO.File]::WriteAllText(
+       "C:\ProgramData\ssh\administrators_authorized_keys",
+       "ssh-ed25519 AAAA... user@hostname`n",
+       [System.Text.UTF8Encoding]::new($false)
+   )
+   ```
+
+3. **Lock down permissions** — OpenSSH requires strict ACLs:
+   ```powershell
+   icacls "C:\ProgramData\ssh\administrators_authorized_keys" /inheritance:r /grant "SYSTEM:(F)" /grant "Administrators:(F)"
+   ```
+
+4. **Restart sshd:**
+   ```powershell
+   Restart-Service sshd
+   ```
+
+### Troubleshooting
+
+- If key auth silently fails, check `Get-WinEvent -LogName OpenSSH/Operational -MaxEvents 10`
+- Common cause: BOM in the key file or `sshd_config` — PowerShell file-writing commands are the usual culprit
+- If the log says `User not allowed because shell does not exist`, the `DefaultShell` registry path is wrong — see [WSL default shell troubleshooting](../../05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md)
+
 ## See Also
 
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../../02-selfhosting/security/linux-server-hardening-checklist.md)
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../process-management/managing-linux-services-systemd-ansible.md)

01-linux/packages/package-management-reference.md

@@ -168,5 +168,5 @@ Flatpak is what I prefer — better sandboxing story, Flathub has most things yo
 
 ## See Also
 
-- [[linux-distro-guide-beginners]]
+- [linux-distro-guide-beginners](../distro-specific/linux-distro-guide-beginners.md)
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../../02-selfhosting/security/linux-server-hardening-checklist.md)

01-linux/process-management/managing-linux-services-systemd-ansible.md

@@ -146,5 +146,5 @@ ansible-playbook -i inventory.ini manage-services.yml
 
 ## See Also
 
-- [[wsl2-instance-migration-fedora43]]
+- [wsl2-instance-migration-fedora43](../distro-specific/wsl2-instance-migration-fedora43.md)
-- [[tuning-netdata-web-log-alerts]]
+- [tuning-netdata-web-log-alerts](../../02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md)

01-linux/shell-scripting/ansible-getting-started.md

@@ -204,5 +204,5 @@ Roles keep things organized and reusable across projects.
 
 ## See Also
 
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../process-management/managing-linux-services-systemd-ansible.md)
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../../02-selfhosting/security/linux-server-hardening-checklist.md)

01-linux/shell-scripting/bash-scripting-patterns.md

@@ -211,5 +211,5 @@ retry 3 10 curl -f https://example.com/health
 
 ## See Also
 
-- [[ansible-getting-started]]
+- [ansible-getting-started](ansible-getting-started.md)
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../process-management/managing-linux-services-systemd-ansible.md)

01-linux/storage/mdadm-raid-rebuild.md

@@ -0,0 +1,113 @@
+---
+title: "mdadm — Rebuilding a RAID Array After Reinstall"
+domain: linux
+category: storage
+tags: [mdadm, raid, linux, storage, recovery, homelab]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+
+# mdadm — Rebuilding a RAID Array After Reinstall
+
+If you reinstall the OS on a machine that has an existing mdadm RAID array, the array metadata is still on the disks — you just need to reassemble it. The data isn't gone unless you've overwritten the member disks.
+
+## The Short Answer
+
+```bash
+# Scan for existing arrays
+sudo mdadm --assemble --scan
+
+# Check what was found
+cat /proc/mdstat
+```
+
+If that works, your array is back. If not, you'll need to manually identify the member disks and reassemble.
+
+## Step-by-Step Recovery
+
+### 1. Identify the RAID member disks
+
+```bash
+# Show mdadm superblock info on each disk/partition
+sudo mdadm --examine /dev/sda1
+sudo mdadm --examine /dev/sdb1
+
+# Or scan all devices at once
+sudo mdadm --examine --scan
+```
+
+Look for matching `UUID` fields — disks with the same array UUID belong to the same array.
+
+### 2. Reassemble the array
+
+```bash
+# Assemble from specific devices
+sudo mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1
+
+# Or let mdadm figure it out from superblocks
+sudo mdadm --assemble --scan
+```
+
+### 3. Verify the array state
+
+```bash
+cat /proc/mdstat
+sudo mdadm --detail /dev/md0
+```
+
+You want to see `State : active` (or `active, degraded` if a disk is missing). If degraded, the array is still usable but should be rebuilt.
+
+### 4. Update mdadm.conf so it persists across reboots
+
+```bash
+# Generate the config
+sudo mdadm --detail --scan | sudo tee -a /etc/mdadm.conf
+
+# Fedora/RHEL — rebuild initramfs so the array is found at boot
+sudo dracut --force
+
+# Debian/Ubuntu — update initramfs
+sudo update-initramfs -u
+```
+
+### 5. Mount the filesystem
+
+```bash
+# Check the filesystem
+sudo fsck /dev/md0
+
+# Mount
+sudo mount /dev/md0 /mnt/raid
+
+# Add to fstab for auto-mount
+echo '/dev/md0 /mnt/raid ext4 defaults 0 2' | sudo tee -a /etc/fstab
+```
+
+## Rebuilding a Degraded Array
+
+If a disk failed or was replaced:
+
+```bash
+# Add the new disk to the existing array
+sudo mdadm --manage /dev/md0 --add /dev/sdc1
+
+# Watch the rebuild progress
+watch cat /proc/mdstat
+```
+
+Rebuild time depends on array size and disk speed. The array is usable during rebuild but with degraded performance.
+
+## Gotchas & Notes
+
+- **Don't `--create` when you mean `--assemble`.** `--create` initializes a new array and will overwrite existing superblocks. `--assemble` brings an existing array back online.
+- **Superblock versions matter.** Modern mdadm uses 1.2 superblocks by default. If the array was created with an older version, specify `--metadata=0.90` during assembly.
+- **RAID is not a backup.** mdadm protects against disk failure, not against accidental deletion, ransomware, or filesystem corruption. Pair it with rsync or Restic for actual backups.
+- **Check SMART status on all member disks** after a reinstall. If you're reassembling because a disk failed, make sure the remaining disks are healthy.
+
+Reference: [mdadm — How to rebuild RAID array after fresh install (Unix & Linux Stack Exchange)](https://unix.stackexchange.com/questions/593836/mdadm-how-to-rebuild-raid-array-after-fresh-install)
+
+## See Also
+
+- [snapraid-mergerfs-setup](snapraid-mergerfs-setup.md)
+- [rsync-backup-patterns](../../02-selfhosting/storage-backup/rsync-backup-patterns.md)

01-linux/storage/snapraid-mergerfs-setup.md

@@ -1,3 +1,12 @@
+---
+title: "SnapRAID & MergerFS Storage Setup"
+domain: linux
+category: storage
+tags: [snapraid, mergerfs, storage, parity, raid, majorraid]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # SnapRAID & MergerFS Storage Setup
 
 ## Problem
@@ -16,19 +25,24 @@ A combination of **MergerFS** for pooling and **SnapRAID** for parity. This is i
 ### 2. Implementation Strategy
 
 1. **Clean the Pool:** Use `rmlint` to clear duplicates and reclaim space.
-2. **Identify the Parity Drive:** Choose your largest drive (or one equal to the largest data drive) to hold the parity information. In my setup, `/mnt/usb` (sdc) was cleared of 4TB of duplicates to be repurposed for this.
+2. **Identify the Parity Drive:** Choose your largest drive (or one equal to the largest data drive) to hold the parity information.
-3. **Configure MergerFS:** Pool the data drives (e.g., `/mnt/disk1`, `/mnt/disk2`) into `/storage`.
+3. **Configure MergerFS:** Pool the data drives into a single mount point.
 4. **Configure SnapRAID:** Point SnapRAID to the data drives and the parity drive.
 
 ### 3. MergerFS Config (/etc/fstab)
 
+On majorhome, the pool mounts three ext4 drives to `/majorRAID`:
+
 ```fstab
-# Example MergerFS pool
+/mnt/disk1:/mnt/disk2:/mnt/disk3 /majorRAID fuse.mergerfs defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=20G,fsname=mergerfsPool 0 0
-/mnt/disk*:/mnt/usb-data /storage fuse.mergerfs defaults,allow_other,cache.files=off,use_ino,category.create=mfs,minfreespace=20G,fsname=mergerfsPool 0 0
 ```
 
+Adjust the source paths and mount point to match your setup. Each `/mnt/diskN` is an individual ext4 drive mounted separately — MergerFS unions them into the single `/majorRAID` path.
+
 ### 4. SnapRAID Config (/etc/snapraid.conf)
 
+> **Note:** SnapRAID is not yet active on majorhome — a 12TB parity drive purchase is deferred. The config below is the planned setup.
+
 ```conf
 # Parity file location
 parity /mnt/parity/snapraid.parity
@@ -37,9 +51,11 @@ parity /mnt/parity/snapraid.parity
 content /var/snapraid/snapraid.content
 content /mnt/disk1/.snapraid.content
 content /mnt/disk2/.snapraid.content
+content /mnt/disk3/.snapraid.content
 
 data d1 /mnt/disk1/
 data d2 /mnt/disk2/
+data d3 /mnt/disk3/
 
 # Exclusions
 exclude /lost+found/
@@ -68,7 +84,3 @@ snapraid scrub
 ```
 
 ---
-
-## Tags
-
-#snapraid #mergerfs #linux #storage #homelab #raid

02-selfhosting/dns-networking/network-overview.md

@@ -1,21 +1,30 @@
+---
+title: "Network Overview"
+domain: selfhosting
+category: dns-networking
+tags: [tailscale, networking, infrastructure, dns, vpn]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # 🌐 Network Overview
 
-The **[[MajorInfrastructure|MajorsHouse]]** infrastructure is connected via a private **[[Network Overview#Tailscale|Tailscale]]** mesh network. This allows secure, peer-to-peer communication between devices across different geographic locations (US and UK) without exposing services to the public internet.
+The **MajorsHouse** infrastructure is connected via a private **Tailscale** mesh network. This allows secure, peer-to-peer communication between devices across different geographic locations (US and UK) without exposing services to the public internet.
 
 ## 🏛️ Infrastructure Summary
 
 - **Address Space:** 100.x.x.x (Tailscale CGNAT)
-- **Management:** Centralized via **[[Network Overview#Ansible|Ansible]]** (`MajorAnsible` repo)
+- **Management:** Centralized via **Ansible** (`MajorAnsible` repo)
 - **Host Groupings:** Functional (web, mail, homelab, bots), OS (Fedora, Ubuntu), and Location (US, UK).
 
 ## 🌍 Geographic Nodes
 
 | Host | Location | IP | OS |
 |---|---|---|---|
-| `[[dca|dca]]` | 🇺🇸 US | 100.104.11.146 | Ubuntu 24.04 |
+| `dcaprod` | 🇺🇸 US | 100.104.11.146 | Ubuntu 24.04 |
-| `[[majortoot|majortoot]]` | 🇺🇸 US | 100.110.197.17 | Ubuntu 24.04 |
+| `majortoot` | 🇺🇸 US | 100.110.197.17 | Ubuntu 24.04 |
-| `[[majorhome|majorhome]]` | 🇺🇸 US | 100.120.209.106 | Fedora 43 |
+| `majorhome` | 🇺🇸 US | 100.120.209.106 | Fedora 43 |
-| `[[teelia|teelia]]` | 🇬🇧 UK | 100.120.32.69 | Ubuntu 24.04 |
+| `teelia` | 🇬🇧 UK | 100.120.32.69 | Ubuntu 24.04 |
 
 ## 🔗 Tailscale Setup
 

02-selfhosting/dns-networking/tailscale-homelab-remote-access.md

@@ -140,6 +140,6 @@ Now any device on your home LAN is reachable from anywhere on the tailnet, even
 
 ## See Also
 
-- [[self-hosting-starter-guide]]
+- [self-hosting-starter-guide](../docker/self-hosting-starter-guide.md)
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../security/linux-server-hardening-checklist.md)
-- [[setting-up-caddy-reverse-proxy]]
+- [setting-up-caddy-reverse-proxy](../reverse-proxy/setting-up-caddy-reverse-proxy.md)

02-selfhosting/docker/debugging-broken-docker-containers.md

@@ -164,5 +164,5 @@ Don't jump straight to the nuclear option. Only use `-v` if you want a completel
 
 ## See Also
 
-- [[docker-vs-vms-homelab]]
+- [docker-vs-vms-homelab](docker-vs-vms-homelab.md)
-- [[tuning-netdata-web-log-alerts]]
+- [tuning-netdata-web-log-alerts](../monitoring/tuning-netdata-web-log-alerts.md)

02-selfhosting/docker/docker-healthchecks.md

@@ -153,5 +153,5 @@ healthcheck:
 
 ## See Also
 
-- [[debugging-broken-docker-containers]]
+- [debugging-broken-docker-containers](debugging-broken-docker-containers.md)
-- [[netdata-docker-health-alarm-tuning]]
+- [netdata-docker-health-alarm-tuning](../monitoring/netdata-docker-health-alarm-tuning.md)

02-selfhosting/docker/docker-vs-vms-homelab.md

@@ -91,5 +91,5 @@ The two coexist fine on the same host. Docker handles the service layer, KVM han
 
 ## See Also
 
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../../01-linux/process-management/managing-linux-services-systemd-ansible.md)
-- [[tuning-netdata-web-log-alerts]]
+- [tuning-netdata-web-log-alerts](../monitoring/tuning-netdata-web-log-alerts.md)

02-selfhosting/docker/self-hosting-starter-guide.md

@@ -110,6 +110,6 @@ Tailscale is the easiest and safest starting point for personal use.
 
 ## See Also
 
-- [[docker-vs-vms-homelab]]
+- [docker-vs-vms-homelab](docker-vs-vms-homelab.md)
-- [[debugging-broken-docker-containers]]
+- [debugging-broken-docker-containers](debugging-broken-docker-containers.md)
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../security/linux-server-hardening-checklist.md)

02-selfhosting/docker/watchtower-smtp-localhost-relay.md

@@ -0,0 +1,105 @@
+---
+title: "Watchtower SMTP via Localhost Postfix Relay"
+domain: selfhosting
+category: docker
+tags: [watchtower, docker, smtp, postfix, email, notifications]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+# Watchtower SMTP via Localhost Postfix Relay
+
+## The Problem
+
+Watchtower supports email notifications via its built-in shoutrrr SMTP driver. The typical setup stores SMTP credentials in the compose file or a separate env file. This creates two failure modes:
+
+1. **Password rotation breaks notifications silently.** When you rotate your mail server password, Watchtower keeps running but stops sending emails. You only discover it when you notice container updates happened with no notification.
+2. **Credentials at rest.** `docker-compose.yml` and `.env` files are often world-readable or checked into git. SMTP passwords stored there are a credential leak waiting to happen.
+
+The shoutrrr SMTP driver also has a quirk: it attempts AUTH over an unencrypted connection to remote SMTP servers, which most mail servers (correctly) reject with `535 5.7.8 authentication failed` or similar.
+
+## The Solution
+
+Route Watchtower's outbound mail through **localhost port 25** using `network_mode: host`. The local Postfix MTA — already running on the host for relay purposes — handles authentication to the upstream mail server. Watchtower never sees a credential.
+
+```
+Watchtower → localhost:25 (Postfix, trusted via mynetworks — no auth required)
+  → Postfix → upstream mail server → delivery
+```
+
+## docker-compose.yml
+
+```yaml
+services:
+  watchtower:
+    image: containrrr/watchtower
+    restart: always
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+    environment:
+      - DOCKER_API_VERSION=1.44
+      - WATCHTOWER_CLEANUP=true
+      - WATCHTOWER_SCHEDULE=0 0 4 * * *
+      - WATCHTOWER_INCLUDE_STOPPED=false
+      - WATCHTOWER_NOTIFICATIONS=email
+      - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
+      - WATCHTOWER_NOTIFICATION_EMAIL_TO=you@yourdomain.com
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=localhost
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=25
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY=true
+      - WATCHTOWER_NOTIFICATION_EMAIL_DELAY=2
+```
+
+**Key settings:**
+- `network_mode: host` — required so `localhost` resolves to the host's loopback interface (and port 25). Without this, `localhost` resolves to the container's own loopback, which has no Postfix.
+- `EMAIL_SERVER=localhost`, `PORT=25` — target the local Postfix
+- `TLS_SKIP_VERIFY=true` — shoutrrr still negotiates STARTTLS even on port 25; a self-signed or expired local Postfix cert is fine to skip
+- No `EMAIL_SERVER_USER` or `EMAIL_SERVER_PASSWORD` — Postfix trusts `127.0.0.1` via `mynetworks`, no auth needed
+
+## Prerequisites
+
+The host needs a Postfix instance that:
+1. Listens on `localhost:25`
+2. Includes `127.0.0.0/8` in `mynetworks` so local processes can relay without authentication
+3. Is configured to relay outbound to your actual mail server
+
+This is standard for any host already running a Postfix relay. If Postfix isn't installed, a minimal relay-only config is a few lines in `main.cf`.
+
+## Why Not Just Use an Env File?
+
+A separate env file (mode 0600) is better than inline compose, but you still have a credential that breaks on rotation. The localhost relay pattern eliminates the credential entirely.
+
+| Approach | Credentials stored | Rotation-safe |
+|---|---|---|
+| Inline in compose | Yes (plaintext, often 0644) | ❌ |
+| Separate env file (0600) | Yes (protected but present) | ❌ |
+| Localhost Postfix relay | None | ✅ |
+
+## Testing
+
+After `docker compose up -d`, check the Watchtower logs for a startup notification:
+
+```bash
+docker logs <watchtower-container-name> 2>&1 | head -20
+# Look for: "Sending notification..."
+```
+
+Confirm Postfix delivered it:
+
+```bash
+grep watchtower /var/log/mail.log | tail -5
+# Look for: status=sent (250 2.0.0 Ok)
+```
+
+## Gotchas
+
+- **`network_mode: host` is Linux-only.** Docker Desktop on macOS/Windows doesn't support host networking. This pattern only works on Linux hosts.
+- **`network_mode: host` drops port mappings.** Any `ports:` entries are silently ignored under `network_mode: host`. Watchtower doesn't expose ports, so this isn't an issue.
+- **Postfix TLS cert warning.** shoutrrr attempts STARTTLS on port 25 regardless. If the local Postfix has a self-signed or expired cert, `TLS_SKIP_VERIFY=true` suppresses the error. For a proper fix, renew the Postfix cert.
+- **`WATCHTOWER_DISABLE_CONTAINERS`.** If you run stacks that manage their own updates (Nextcloud AIO, etc.), list those containers here (space-separated) to prevent Watchtower from interfering.
+
+## See Also
+
+- [docker-healthchecks](docker-healthchecks.md)
+- [debugging-broken-docker-containers](debugging-broken-docker-containers.md)

02-selfhosting/index.md

@@ -1,3 +1,7 @@
+---
+created: 2026-04-13T10:15
+updated: 2026-04-13T10:15
+---
 # 🏠 Self-Hosting & Homelab
 
 Guides for running your own services at home, including Docker, reverse proxies, DNS, storage, monitoring, and security.
@@ -29,3 +33,9 @@ Guides for running your own services at home, including Docker, reverse proxies,
 ## Security
 
 - [Linux Server Hardening Checklist](security/linux-server-hardening-checklist.md)
+- [Standardizing unattended-upgrades with Ansible](security/ansible-unattended-upgrades-fleet.md)
+- [Fail2ban Custom Jail: Apache 404 Scanner Detection](security/fail2ban-apache-404-scanner-jail.md)
+- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](security/fail2ban-apache-php-probe-jail.md)
+- [Fail2ban Custom Jail: WordPress Login Brute Force](security/fail2ban-wordpress-login-jail.md)
+- [SELinux: Fixing Fail2ban grep execmem Denial](security/selinux-fail2ban-execmem-fix.md)
+- [UFW Firewall Management](security/ufw-firewall-management.md)

02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md

@@ -5,7 +5,7 @@ category: monitoring
 tags: [netdata, docker, nextcloud, alarms, health, monitoring]
 status: published
 created: 2026-03-18
-updated: 2026-03-22
+updated: 2026-03-28
 ---
 
 # Tuning Netdata Docker Health Alarms to Prevent Update Flapping
@@ -28,8 +28,13 @@ A single container being unhealthy for 10 seconds triggers it. No grace period,
 
 Create a custom override at `/etc/netdata/health.d/docker.conf` (maps to the Netdata config volume if running in Docker). This file takes precedence over the stock config in `/usr/lib/netdata/conf.d/health.d/docker.conf`.
 
+### General Container Alarm
+
+This alarm covers all containers **except** `nextcloud-aio-nextcloud`, which gets its own dedicated alarm (see below).
+
 ```ini
 # Custom override — reduces flapping during nightly container updates.
+# General container unhealthy alarm — all containers except nextcloud-aio-nextcloud
 
 template: docker_container_unhealthy
        on: docker.container_health_status
@@ -39,6 +44,7 @@ component: Docker
     units: status
     every: 30s
    lookup: average -5m of unhealthy
+chart labels: container_name=!nextcloud-aio-nextcloud *
      warn: $this > 0
     delay: up 3m down 5m multiplier 1.5 max 30m
   summary: Docker container ${label:container_name} health
@@ -53,7 +59,47 @@ component: Docker
 | `delay: up 3m` | none | 3m | Won't fire until unhealthy condition persists for 3 continuous minutes |
 | `delay: down 5m` | none | 5m (max 30m) | Grace period after recovery before clearing |
 
-The `up` delay is the critical addition. Nextcloud AIO's `nextcloud-aio-nextcloud` container checks both PostgreSQL (port 5432) and PHP-FPM (port 9000). PHP-FPM takes ~90 seconds to warm up after a restart, causing 2–3 failing health checks before the container becomes healthy. With `delay: up 3m`, Netdata waits for 3 continuous minutes of unhealthy status before firing — absorbing the ~90 second startup window with margin to spare. A genuinely broken container will still trigger the alert.
+### Dedicated Nextcloud AIO Alarm
+
+Added 2026-03-23, updated 2026-03-28. The `nextcloud-aio-nextcloud` container needs a more lenient window than other containers. Its healthcheck (`/healthcheck.sh`) verifies PostgreSQL connectivity (port 5432) and PHP-FPM (port 9000). PHP-FPM takes ~90 seconds to warm up after a normal restart — but during nightly AIO update cycles, the full startup (occ upgrade, app updates, migrations) can take 5+ minutes. On 2026-03-27, a startup hung and left the container unhealthy for 20 hours until the next nightly cycle replaced it.
+
+The dedicated alarm uses a 10-minute lookup window and 10-minute delay to absorb normal startup, while still catching sustained failures:
+
+```ini
+# Dedicated alarm for nextcloud-aio-nextcloud — lenient window to absorb nightly update cycle
+# PHP-FPM can take 5+ minutes to warm up; only alert on sustained failure
+
+template: docker_nextcloud_unhealthy
+       on: docker.container_health_status
+    class: Errors
+     type: Containers
+component: Docker
+    units: status
+    every: 30s
+   lookup: average -10m of unhealthy
+chart labels: container_name=nextcloud-aio-nextcloud
+     warn: $this > 0
+    delay: up 10m down 5m multiplier 1.5 max 30m
+  summary: Nextcloud container health sustained
+     info: nextcloud-aio-nextcloud has been unhealthy for a sustained period — not a transient update blip
+       to: sysadmin
+```
+
+## Watchdog Cron: Auto-Restart on Sustained Unhealthy
+
+If the Nextcloud container stays unhealthy for more than 1 hour (well past any normal startup window), a cron watchdog on majorlab auto-restarts it and logs the event. This was added 2026-03-28 after an incident where the container sat unhealthy for 20 hours until the next nightly backup cycle replaced it.
+
+**File:** `/etc/cron.d/nextcloud-health-watchdog`
+
+```bash
+# Restart nextcloud-aio-nextcloud if unhealthy for >1 hour
+*/15 * * * * root docker inspect --format={{.State.Health.Status}} nextcloud-aio-nextcloud 2>/dev/null | grep -q unhealthy && [ "$(docker inspect --format={{.State.StartedAt}} nextcloud-aio-nextcloud | xargs -I{} date -d {} +\%s)" -lt "$(date -d "1 hour ago" +\%s)" ] && docker restart nextcloud-aio-nextcloud && logger -t nextcloud-watchdog "Restarted unhealthy nextcloud-aio-nextcloud"
+```
+
+- Runs every 15 minutes as root
+- Only restarts if the container has been running for >1 hour (avoids interfering with normal startup)
+- Logs to syslog as `nextcloud-watchdog` — check with `journalctl -t nextcloud-watchdog`
+- Netdata will still fire the `docker_nextcloud_unhealthy` alert during the unhealthy window, but the outage is capped at ~1 hour instead of persisting until the next nightly cycle
 
 ## Also: Suppress `docker_container_down` for Normally-Exiting Containers
 

02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md

@@ -1,3 +1,12 @@
+---
+title: "Netdata n8n Enriched Alert Emails"
+domain: selfhosting
+category: monitoring
+tags: [netdata, n8n, alerts, email, monitoring, automation]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Netdata → n8n Enriched Alert Emails
 
 **Status:** Live across all MajorsHouse fleet servers as of 2026-03-21

02-selfhosting/monitoring/netdata-selinux-avc-chart.md

@@ -134,4 +134,4 @@ done
 - [Deploying Netdata to a New Server](netdata-new-server-setup.md)
 - [Tuning Netdata Web Log Alerts](tuning-netdata-web-log-alerts.md)
 - [Tuning Netdata Docker Health Alarms](netdata-docker-health-alarm-tuning.md)
-- [SELinux: Fixing Dovecot Mail Spool Context](/05-troubleshooting/selinux-dovecot-vmail-context.md)
+- [SELinux: Fixing Dovecot Mail Spool Context](../../05-troubleshooting/selinux-dovecot-vmail-context.md)

02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md

@@ -85,4 +85,4 @@ curl -s http://localhost:19999/api/v1/alarms?all | grep -A 15 "web_log_1m_redire
 
 ## See Also
 
-- [[Netdata service monitoring]]
+- Netdata service monitoring

02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md

@@ -135,6 +135,6 @@ yourdomain.com {
 
 ## See Also
 
-- [[self-hosting-starter-guide]]
+- [self-hosting-starter-guide](../docker/self-hosting-starter-guide.md)
-- [[linux-server-hardening-checklist]]
+- [linux-server-hardening-checklist](../security/linux-server-hardening-checklist.md)
-- [[debugging-broken-docker-containers]]
+- [debugging-broken-docker-containers](../docker/debugging-broken-docker-containers.md)

02-selfhosting/security/ansible-unattended-upgrades-fleet.md

@@ -90,5 +90,5 @@ ansible-playbook update.yml -l dca,majorlinux,majortoot
 
 ## See Also
 
-- [[ansible-getting-started|Ansible Getting Started]]
+- [Ansible Getting Started](../../01-linux/shell-scripting/ansible-getting-started.md)
-- [[linux-server-hardening-checklist|Linux Server Hardening Checklist]]
+- [Linux Server Hardening Checklist](linux-server-hardening-checklist.md)

02-selfhosting/security/fail2ban-apache-404-scanner-jail.md

@@ -0,0 +1,127 @@
+---
+title: "Fail2ban Custom Jail: Apache 404 Scanner Detection"
+domain: selfhosting
+category: security
+tags: [fail2ban, apache, security, scanner, firewall]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# Fail2ban Custom Jail: Apache 404 Scanner Detection
+
+## The Problem
+
+Automated vulnerability scanners probe web servers by requesting dozens of common config file paths — `.env`, `env.php`, `next.config.js`, `nuxt.config.ts`, etc. — in rapid succession. These all return **404 Not Found**, which is correct behavior from Apache.
+
+However, the built-in Fail2ban jails (`apache-noscript`, `apache-botsearch`) don't catch these because they parse the **error log**, not the **access log**. If Apache doesn't write a corresponding "File does not exist" entry to the error log for every 404, the scanner slips through undetected.
+
+This also triggers false alerts in monitoring tools like **Netdata**, which sees the success ratio drop (e.g., `web_log_1m_successful` goes CRITICAL at 2.83%) because 404s aren't counted as successful responses.
+
+## The Solution
+
+Create a custom Fail2ban filter that reads the **access log** and matches 404 responses directly.
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/apache-404scan.conf`:
+
+```ini
+# Fail2Ban filter to catch rapid 404 scanning in Apache access logs
+# Targets vulnerability scanners probing for .env, config files, etc.
+
+[Definition]
+
+# Match 404 responses in combined/common access log format
+failregex = ^<HOST> -.*"(GET|POST|HEAD|PUT|DELETE|OPTIONS|PATCH) .+" 404 \d+
+
+ignoreregex = ^<HOST> -.*(robots\.txt|favicon\.ico|apple-touch-icon)
+
+datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
+```
+
+### Step 2 — Add the jail
+
+Add to `/etc/fail2ban/jail.local`:
+
+```ini
+[apache-404scan]
+enabled = true
+port = http,https
+filter = apache-404scan
+logpath = /var/log/apache2/access.log
+maxretry = 10
+findtime = 1m
+bantime = 24h
+backend = polling
+```
+
+**10 hits in 1 minute** is aggressive enough to catch scanners (which fire 30–50+ requests in seconds) while avoiding false positives from a legitimate user hitting a few broken links.
+
+> **Critical: `backend = polling` is required** if your `jail.local` or `jail.d/` sets `backend = systemd` in `[DEFAULT]` (common on Fedora/RHEL). Without it, fail2ban ignores the `logpath` and reads from journald instead — which Apache doesn't write to. The jail will appear active (`fail2ban-client status` shows it running) but `fail2ban-client get apache-404scan logpath` will return "No file is currently monitored" and zero IPs will ever be banned. This fails silently.
+
+### Step 3 — Test the regex
+
+```bash
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404scan.conf
+```
+
+You should see matches. In a real-world test against a server under active scanning, this matched **2831 out of 8901** access log lines.
+
+### Step 4 — Reload Fail2ban
+
+```bash
+systemctl restart fail2ban
+fail2ban-client status apache-404scan
+```
+
+## Why Default Jails Miss This
+
+| Jail | Log Source | What It Matches | Why It Misses |
+|---|---|---|---|
+| `apache-noscript` | error log | "script not found or unable to stat" | Only matches script-type files (.php, .asp, .exe, .pl) |
+| `apache-botsearch` | error log | "File does not exist" for specific paths | Requires Apache to write error log entries for 404s |
+| **`apache-404scan`** | **access log** | **Any 404 response** | **Catches everything** |
+
+The key insight: URL-encoded probes like `/%2f%2eenv%2econfig` that return 404 in the access log may not generate error log entries at all, making them invisible to the default filters.
+
+## Pair With Recidive
+
+If you have the `recidive` jail enabled, repeat offenders get permanently banned:
+
+```ini
+[recidive]
+enabled = true
+bantime = -1
+findtime = 86400
+maxretry = 3
+```
+
+Three 24-hour bans within a day = permanent firewall block.
+
+## Quick Diagnostic Commands
+
+```bash
+# Test filter against current access log
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404scan.conf
+
+# Check jail status and banned IPs
+fail2ban-client status apache-404scan
+
+# IMPORTANT: verify the jail is actually monitoring the file
+fail2ban-client get apache-404scan logpath
+# Should show: /var/log/apache2/access.log
+# If it shows "No file is currently monitored" — add backend = polling to the jail
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep apache-404scan
+
+# Count 404s in today's log
+grep '" 404 ' /var/log/apache2/access.log | wc -l
+```
+
+## Key Notes
+
+- The `ignoreregex` excludes `robots.txt`, `favicon.ico`, and `apple-touch-icon` — these are commonly requested and produce harmless 404s.
+- Make sure your Tailscale subnet (`100.64.0.0/10`) is in the `ignoreip` list under `[DEFAULT]` so you don't ban your own monitoring or uptime checks.
+- This filter works with both Apache **combined** and **common** log formats.
+- Complements the existing `apache-dirscan` jail (which catches error-log-based directory enumeration). Use both for full coverage.

02-selfhosting/security/fail2ban-apache-bad-request-jail.md

@@ -0,0 +1,127 @@
+---
+title: "Fail2ban Custom Jail: Apache Bad Request Detection"
+domain: selfhosting
+category: security
+tags: [fail2ban, apache, security, firewall, bad-request]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+# Fail2ban Custom Jail: Apache Bad Request Detection
+
+## The Problem
+
+fail2ban ships a stock `nginx-bad-request` filter for catching malformed HTTP requests (400s), but **there is no Apache equivalent**. Apache servers are left unprotected against the same class of attack: scanners that send garbage request lines to probe for vulnerabilities or overwhelm the access log.
+
+Unlike the nginx version, this filter has to be written from scratch.
+
+## The Solution
+
+Create a custom filter targeting **400 Bad Request** responses in Apache's Combined Log Format, then wire it to a jail.
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/apache-bad-request.conf`:
+
+```ini
+# Fail2Ban filter: catch 400 Bad Request responses in Apache access logs
+# Targets malformed HTTP requests — garbage request lines, empty methods, etc.
+# No stock equivalent exists; nginx-bad-request ships with fail2ban but Apache does not.
+
+[Definition]
+
+# Match 400 responses in Apache Combined/Common Log Format
+failregex = ^<HOST> -.*".*" 400 \d+
+
+ignoreregex =
+
+datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
+```
+
+### Step 2 — Validate the filter
+
+Always test before deploying:
+
+```bash
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf
+```
+
+Against a live server under typical traffic this matched **155 lines with zero false positives**. If you see unexpected matches, refine `ignoreregex`.
+
+### Step 3 — Create the jail drop-in
+
+Create `/etc/fail2ban/jail.d/apache-bad-request.conf`:
+
+```ini
+[apache-bad-request]
+enabled  = true
+port     = http,https
+filter   = apache-bad-request
+logpath  = /var/log/apache2/access.log
+maxretry = 10
+findtime = 60
+bantime  = 1h
+```
+
+> **Note:** On Fedora/RHEL, the log path may be `/var/log/httpd/access_log`. If your `[DEFAULT]` sets `backend = systemd`, add `backend = polling` to the jail — otherwise it silently ignores `logpath` and reads journald instead.
+
+### Step 4 — Reload fail2ban
+
+```bash
+systemctl reload fail2ban
+fail2ban-client status apache-bad-request
+```
+
+## Deploy Fleet-Wide with Ansible
+
+If you run multiple Apache hosts, use Ansible to deploy both the filter and jail atomically:
+
+```yaml
+- name: Deploy apache-bad-request fail2ban filter
+  ansible.builtin.template:
+    src: templates/fail2ban_apache_bad_request_filter.conf.j2
+    dest: /etc/fail2ban/filter.d/apache-bad-request.conf
+  notify: Reload fail2ban
+
+- name: Deploy apache-bad-request fail2ban jail
+  ansible.builtin.template:
+    src: templates/fail2ban_apache_bad_request_jail.conf.j2
+    dest: /etc/fail2ban/jail.d/apache-bad-request.conf
+  notify: Reload fail2ban
+```
+
+## Why Not Use nginx-bad-request on Apache?
+
+The `nginx-bad-request` filter parses nginx's log format, which differs from Apache's Combined Log Format. The timestamp format, field ordering, and quoting differ enough that the regex won't match. You need a separate filter.
+
+| | nginx-bad-request | apache-bad-request |
+|---|---|---|
+| Ships with fail2ban | ✅ Yes | ❌ No — must write custom |
+| Log source | nginx access log | Apache access log |
+| What it catches | 400 responses (malformed requests) | 400 responses (malformed requests) |
+| Regex target | nginx Combined Log Format | Apache Combined Log Format |
+
+## Diagnostic Commands
+
+```bash
+# Validate filter against live log
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf
+
+# Check jail status
+fail2ban-client status apache-bad-request
+
+# Confirm the jail is monitoring the correct log file
+fail2ban-client get apache-bad-request logpath
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep apache-bad-request
+
+# Count 400s in today's access log
+grep '" 400 ' /var/log/apache2/access.log | wc -l
+```
+
+## See Also
+
+- [fail2ban-nginx-bad-request-jail](fail2ban-nginx-bad-request-jail.md) — the nginx equivalent (stock filter, just needs wiring)
+- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md) — catches 404 probe scanners
+- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md)

02-selfhosting/security/fail2ban-apache-php-probe-jail.md

@@ -0,0 +1,146 @@
+---
+title: "Fail2ban Custom Jail: Apache PHP Webshell Probe Detection"
+domain: selfhosting
+category: security
+tags:
+  - fail2ban
+  - apache
+  - security
+  - php
+  - webshell
+  - scanner
+status: published
+created: 2026-04-09
+updated: 2026-04-13T10:15
+---
+# Fail2ban Custom Jail: Apache PHP Webshell Probe Detection
+
+## The Problem
+
+Automated scanners flood web servers with rapid-fire requests for non-existent `.php` files — `bless.php`, `alfa.php`, `lock360.php`, `about.php`, `cgi-bin/bypass.php`, and hundreds of others. These are classic **webshell/backdoor probes** looking for compromised PHP files left behind by prior attackers.
+
+On servers that force HTTPS (or have HTTP→HTTPS redirects in place), these probes often return **301 Moved Permanently** instead of 404. That causes three problems:
+
+1. **The `apache-404scan` jail misses them** — it only matches 404 responses
+2. **Netdata fires false `web_log_1m_redirects` alerts** — the redirect ratio spikes to 96%+ during scans
+3. **The scanner is never banned**, and will return repeatedly
+
+This was the exact trigger for the 2026-04-09 `[MajorLinux] Web Log Alert` incident where `45.86.202.224` sent 202 PHP probe requests in a few minutes, all returning 301.
+
+## The Solution
+
+Create a custom Fail2ban filter that matches **any `.php` request returning a redirect, forbidden, or not-found response** — while excluding legitimate WordPress PHP endpoints.
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/apache-php-probe.conf`:
+
+```ini
+# Fail2Ban filter to catch PHP file probing (webshell/backdoor scanners)
+# These requests hit non-existent .php files and get 301/302/403/404 responses
+
+[Definition]
+
+failregex = ^<HOST> -.*"(GET|POST|HEAD) /[^ ]*\.php[^ ]* HTTP/[0-9.]+" (301|302|403|404) \d+
+
+ignoreregex = ^<HOST> -.*(wp-cron\.php|xmlrpc\.php|wp-login\.php|wp-admin|index\.php|wp-comments-post\.php)
+
+datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
+```
+
+**Why the ignoreregex matters:** Legitimate WordPress traffic hits `wp-cron.php`, `xmlrpc.php` (often 403-blocked on hardened sites), `wp-login.php`, and `index.php` constantly. Without exclusions the jail would ban your own WordPress admins. Note that `wp-login.php` brute force is caught separately by the `wordpress` jail.
+
+### Step 2 — Add the jail
+
+Add to `/etc/fail2ban/jail.local`:
+
+```ini
+[apache-php-probe]
+enabled = true
+port = http,https
+filter = apache-php-probe
+logpath = /var/log/apache2/access.log
+maxretry = 5
+findtime = 1m
+bantime = 48h
+```
+
+**5 hits in 1 minute** is tight — scanners fire 20–200 PHP probes in seconds, while a real user hitting one broken PHP link won't trip the threshold. The 48-hour bantime is longer than `apache-404scan`'s 24h because PHP webshell scanning is a stronger signal of malicious intent.
+
+### Step 3 — Test the regex
+
+```bash
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-php-probe.conf
+```
+
+Verify it matches the scanner requests and does **not** match legitimate WordPress traffic.
+
+### Step 4 — Reload Fail2ban
+
+```bash
+systemctl restart fail2ban
+fail2ban-client status apache-php-probe
+```
+
+## Why This Complements `apache-404scan`
+
+| Jail | Catches | Misses |
+|---|---|---|
+| `apache-404scan` | Any 404 (config file probes, `.env`, random paths) | PHP probes redirected to HTTPS (301) |
+| **`apache-php-probe`** | **PHP webshell probes (301/302/403/404)** | Non-`.php` probes |
+
+Running both jails together covers:
+- **HTTP→HTTPS redirected PHP probes** (301 responses)
+- **Directly-served PHP probes** (404 responses)
+- **Blocked PHP paths** like `xmlrpc.php` in non-WP contexts (403 responses)
+
+## Pair With Recidive
+
+The `recidive` jail catches repeat offenders across all jails:
+
+```ini
+[recidive]
+enabled = true
+bantime = -1
+findtime = 86400
+maxretry = 3
+```
+
+A scanner that trips `apache-php-probe` three times in 24 hours gets a **permanent** firewall-level ban.
+
+## Manual IP Blocking via UFW
+
+For known scanners you want to block immediately without waiting for the jail to trip, use UFW:
+
+```bash
+# Insert at top of rule list (priority over Apache ALLOW rules)
+ufw insert 1 deny from <IP> to any comment "PHP webshell scanner YYYY-MM-DD"
+```
+
+This bypasses fail2ban entirely and is useful for:
+- Scanners you spot in logs after the fact
+- Known-malicious subnets from threat intel
+- Entire CIDR blocks (`ufw insert 1 deny from 45.86.202.0/24`)
+
+## Quick Diagnostic Commands
+
+```bash
+# Count recent PHP probes returning 301/403/404
+awk '/09\/Apr\/2026:18:/ && /\.php/ && ($9==301 || $9==403 || $9==404)' /var/log/apache2/access.log | wc -l
+
+# Top probed PHP filenames (useful for writing additional ignoreregex)
+grep '\.php' /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
+
+# Top scanner IPs by PHP probe count
+grep '\.php' /var/log/apache2/access.log | awk '$9 ~ /^(301|403|404)$/ {print $1}' | sort | uniq -c | sort -rn | head -10
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep apache-php-probe
+```
+
+## Key Notes
+
+- **This jail only makes sense on servers that redirect HTTP→HTTPS.** On plain-HTTPS-only servers, PHP probes return 404 and `apache-404scan` already catches them.
+- **Add your own WordPress plugin paths to `ignoreregex`** if you use non-standard endpoints (e.g., custom admin URLs, REST API `.php` handlers).
+- **This filter pairs naturally with Netdata `web_log_1m_redirects` alerts** — during a scan, Netdata fires first (threshold crossed), then fail2ban bans the IP within seconds.
+- Also see: [Fail2ban Custom Jail: Apache 404 Scanner Detection](fail2ban-apache-404-scanner-jail.md) for the sibling 404-based filter.

02-selfhosting/security/fail2ban-nginx-bad-request-jail.md

@@ -0,0 +1,89 @@
+---
+title: "Fail2ban: Enable the nginx-bad-request Jail"
+domain: selfhosting
+category: security
+tags: [fail2ban, nginx, security, firewall, bad-request]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+# Fail2ban: Enable the nginx-bad-request Jail
+
+## The Problem
+
+Automated scanners sometimes send **malformed HTTP requests** — empty request lines, truncated headers, or garbage data — that nginx rejects with a `400 Bad Request`. These aren't caught by the default fail2ban jails (`nginx-botsearch`, `nginx-http-auth`) because those target URL-probe patterns and auth failures, not raw protocol abuse.
+
+In a real incident: a single IP (`185.177.72.70`) sent **2,778 malformed requests in ~4 minutes**, driving Netdata's `web_log_1m_bad_requests` to 93.7% and triggering a CRITICAL alert. The neighboring IP (`185.177.72.61`) was already banned — the `/24` was known-bad and operating in shifts.
+
+## The Solution
+
+fail2ban ships a `nginx-bad-request` filter out of the box. It's just not wired to a jail by default. Enabling it is a one-step drop-in.
+
+### Step 1 — Create the jail drop-in
+
+Create `/etc/fail2ban/jail.d/nginx-bad-request.conf`:
+
+```ini
+[nginx-bad-request]
+enabled  = true
+port     = http,https
+filter   = nginx-bad-request
+logpath  = /var/log/nginx/access.log
+maxretry = 10
+findtime = 60
+bantime  = 1h
+```
+
+**Settings rationale:**
+- `maxretry = 10` — a legitimate browser never sends 10 malformed requests; this threshold catches burst scanners immediately
+- `findtime = 60` — 60-second window; the attack pattern fires dozens of requests per minute
+- `bantime = 1h` — reasonable starting point; pair with `recidive` for repeat offenders
+
+### Step 2 — Verify the filter matches your log format
+
+Before reloading, confirm the stock filter matches your nginx logs:
+
+```bash
+fail2ban-regex /var/log/nginx/access.log nginx-bad-request
+```
+
+In a real-world test against an active server this matched **2,829 lines with zero false positives**.
+
+### Step 3 — Reload fail2ban
+
+```bash
+systemctl reload fail2ban
+fail2ban-client status nginx-bad-request
+```
+
+You can also ban an IP manually while the jail is loading:
+
+```bash
+fail2ban-client set nginx-bad-request banip 185.177.72.70
+```
+
+## Verify It's Working
+
+```bash
+# Check jail status and active bans
+fail2ban-client status nginx-bad-request
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep nginx-bad-request
+
+# Confirm the jail is monitoring the right file
+fail2ban-client get nginx-bad-request logpath
+```
+
+## Key Notes
+
+- The stock filter is at `/etc/fail2ban/filter.d/nginx-bad-request.conf` — no need to create it.
+- If your `[DEFAULT]` section sets `backend = systemd` (common on Fedora/RHEL), add `backend = polling` to the jail or it will silently ignore `logpath` and monitor journald instead — where nginx doesn't write.
+- Make sure your Tailscale subnet (`100.64.0.0/10`) is in `ignoreip` under `[DEFAULT]` to avoid banning your own monitoring.
+- This jail targets **400 Bad Request** responses. For 404 scanner detection, see [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md).
+
+## See Also
+
+- [fail2ban-apache-bad-request-jail](fail2ban-apache-bad-request-jail.md) — Apache equivalent (no stock filter; custom filter required)
+- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md)
+- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md)

02-selfhosting/security/fail2ban-wordpress-login-jail.md

@@ -0,0 +1,131 @@
+---
+title: "Fail2ban Custom Jail: WordPress Login Brute Force"
+domain: selfhosting
+category: security
+tags: [fail2ban, wordpress, apache, security, brute-force]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# Fail2ban Custom Jail: WordPress Login Brute Force
+
+## The Problem
+
+WordPress login brute force attacks are extremely common. Bots hammer `/wp-login.php` with POST requests, cycling through common credentials. The default Fail2ban `apache-auth` jail doesn't catch these because WordPress returns **HTTP 200** on failed logins — not 401 — so nothing appears as an authentication failure in the Apache error log.
+
+There are pre-packaged filters (`wordpress-hard.conf`, `wordpress-soft.conf`) that ship with some Fail2ban installations, but these require the **[WP fail2ban](https://wordpress.org/plugins/wp-fail2ban/)** WordPress plugin to be installed. That plugin writes login failures to syslog, which the filters then match. Without the plugin, those filters do nothing.
+
+## The Solution
+
+Create a lightweight filter that reads the **Apache access log** and matches repeated POST requests to `wp-login.php` directly. No WordPress plugin needed.
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/wordpress-login.conf`:
+
+```ini
+# Fail2Ban filter for WordPress login brute force
+# Matches POST requests to wp-login.php in Apache access log
+
+[Definition]
+
+failregex = ^<HOST> .* "POST /wp-login\.php
+
+ignoreregex =
+```
+
+### Step 2 — Add the jail
+
+Add to `/etc/fail2ban/jail.local`:
+
+```ini
+[wordpress-login]
+enabled = true
+port = http,https
+filter = wordpress-login
+logpath = /var/log/apache2/access.log
+maxretry = 5
+findtime = 60
+bantime = 30d
+backend = polling
+```
+
+**5 attempts in 60 seconds** is tight enough to catch bots (which fire hundreds of requests per minute) while giving a real human a reasonable margin for typos.
+
+> **Critical: `backend = polling` is required** on Ubuntu 24.04 and other systemd-based distros where `backend = auto` defaults to `systemd`. Without it, Fail2ban ignores `logpath` and reads from journald, which Apache doesn't write to. The jail silently monitors nothing. See [[fail2ban-apache-404-scanner-jail]] for more detail on this gotcha.
+
+### Step 3 — Test the regex
+
+```bash
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/wordpress-login.conf
+```
+
+In a real-world test against an active brute force (3 IPs, ~1,700 hits each), this matched **5,178 lines**.
+
+### Step 4 — Reload and verify
+
+```bash
+systemctl restart fail2ban
+fail2ban-client status wordpress-login
+```
+
+### Manually banning known attackers
+
+If you've already identified brute-force IPs from the logs, ban them immediately rather than waiting for new hits:
+
+```bash
+# Find top offenders
+grep "POST /wp-login.php" /var/log/apache2/access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10
+
+# Ban them
+fail2ban-client set wordpress-login banip <IP>
+```
+
+## Why Default Jails Miss This
+
+| Jail | Log Source | What It Matches | Why It Misses |
+|---|---|---|---|
+| `apache-auth` | error log | 401 authentication failures | WordPress returns 200, not 401 |
+| `wordpress-hard` | syslog | WP fail2ban plugin messages | Requires plugin installation |
+| `wordpress-soft` | syslog | WP fail2ban plugin messages | Requires plugin installation |
+| **`wordpress-login`** | **access log** | **POST to wp-login.php** | **No plugin needed** |
+
+## Optional: Extend to XML-RPC
+
+WordPress's `xmlrpc.php` is another common brute-force target. To cover both, update the filter:
+
+```ini
+failregex = ^<HOST> .* "POST /wp-login\.php
+            ^<HOST> .* "POST /xmlrpc\.php
+```
+
+## Quick Diagnostic Commands
+
+```bash
+# Test filter against current access log
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/wordpress-login.conf
+
+# Check jail status and banned IPs
+fail2ban-client status wordpress-login
+
+# Verify the jail is reading the correct file
+fail2ban-client get wordpress-login logpath
+
+# Count wp-login POSTs in today's log
+grep "POST /wp-login.php" /var/log/apache2/access.log | wc -l
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep wordpress-login
+```
+
+## Key Notes
+
+- This filter works with both Apache **combined** and **common** log formats.
+- Make sure your Tailscale subnet (`100.64.0.0/10`) is in the `ignoreip` list under `[DEFAULT]` so legitimate admin access isn't banned.
+- The `recidive` jail (if enabled) will escalate repeat offenders — three 30-day bans within a day triggers a 90-day block.
+- Complements the [[fail2ban-apache-404-scanner-jail|Apache 404 Scanner Jail]] for full access-log coverage.
+
+## See Also
+
+- [[fail2ban-apache-404-scanner-jail]] — catches vulnerability scanners via 404 floods
+- [[tuning-netdata-web-log-alerts]] — suppress false Netdata alerts from normal HTTP traffic

02-selfhosting/security/linux-server-hardening-checklist.md

@@ -194,6 +194,38 @@ sudo systemctl disable --now servicename
 
 Common ones to disable on a dedicated server: `avahi-daemon`, `cups`, `bluetooth`.
 
+## 8. Mail Server: SpamAssassin
+
+If you're running Postfix (like on majormail), SpamAssassin filters incoming spam before it hits your mailbox.
+
+**Install (Fedora/RHEL):**
+
+```bash
+sudo dnf install spamassassin
+sudo systemctl enable --now spamassassin
+```
+
+**Integrate with Postfix** by adding a content filter in `/etc/postfix/master.cf`. See the [full setup guide](https://www.davekb.com/browse_computer_tips:spamassassin_with_postfix:txt) for Postfix integration on RedHat-based systems.
+
+**Train the filter with sa-learn:**
+
+SpamAssassin gets better when you feed it examples of spam and ham (legitimate mail):
+
+```bash
+# Train on known spam
+sa-learn --spam /path/to/spam-folder/
+
+# Train on known good mail
+sa-learn --ham /path/to/ham-folder/
+
+# Check what sa-learn knows
+sa-learn --dump magic
+```
+
+Run `sa-learn` periodically against your Maildir to keep the Bayesian filter accurate. The more examples it sees, the fewer false positives and missed spam you'll get.
+
+Reference: [sa-learn documentation](https://spamassassin.apache.org/full/3.0.x/dist/doc/sa-learn.html)
+
 ## Gotchas & Notes
 
 - **Don't lock yourself out.** Test SSH key auth in a second terminal before disabling passwords. Keep the original session open.
@@ -204,5 +236,5 @@ Common ones to disable on a dedicated server: `avahi-daemon`, `cups`, `bluetooth
 
 ## See Also
 
-- [[managing-linux-services-systemd-ansible]]
+- [managing-linux-services-systemd-ansible](../../01-linux/process-management/managing-linux-services-systemd-ansible.md)
-- [[debugging-broken-docker-containers]]
+- [debugging-broken-docker-containers](../docker/debugging-broken-docker-containers.md)

02-selfhosting/security/selinux-fail2ban-execmem-fix.md

@@ -0,0 +1,95 @@
+---
+title: "SELinux: Fixing Fail2ban grep execmem Denial on Fedora"
+domain: selfhosting
+category: security
+tags: [selinux, fail2ban, fedora, execmem, security]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# SELinux: Fixing Fail2ban grep execmem Denial on Fedora
+
+## The Problem
+
+After a reboot on Fedora 43, Netdata fires a `selinux_avc_denials` WARNING alert. The audit log shows:
+
+```
+avc: denied { execmem } for comm="grep"
+scontext=system_u:system_r:fail2ban_t:s0
+tcontext=system_u:system_r:fail2ban_t:s0
+tclass=process permissive=0
+```
+
+Fail2ban spawns `grep` to scan log files when its jails start. SELinux denies `execmem` (executable memory) for processes running in the `fail2ban_t` domain. The `fail2ban-selinux` package does not include this permission.
+
+## Impact
+
+- Fail2ban still functions — the denial affects grep's memory allocation strategy, not its ability to run
+- Netdata will keep alerting on every reboot (fail2ban restarts and triggers the denial)
+- No security risk — this is fail2ban's own grep subprocess, not an external exploit
+
+## The Fix
+
+Create a targeted SELinux policy module that allows `execmem` for `fail2ban_t`:
+
+```bash
+cd /tmp
+
+cat > my-fail2ban-grep.te << "EOF"
+module my-fail2ban-grep 1.0;
+
+require {
+    type fail2ban_t;
+    class process execmem;
+}
+
+allow fail2ban_t self:process execmem;
+EOF
+
+# Compile the module
+checkmodule -M -m -o my-fail2ban-grep.mod my-fail2ban-grep.te
+
+# Package it
+semodule_package -o my-fail2ban-grep.pp -m my-fail2ban-grep.mod
+
+# Install at priority 300 (above default policy)
+semodule -X 300 -i my-fail2ban-grep.pp
+```
+
+## Verifying
+
+Confirm the module is loaded:
+
+```bash
+semodule -l | grep fail2ban-grep
+# Expected: my-fail2ban-grep
+```
+
+Check that no new AVC denials appear after restarting fail2ban:
+
+```bash
+systemctl restart fail2ban
+ausearch -m avc --start recent | grep fail2ban
+# Expected: no output (no new denials)
+```
+
+## Why Not `audit2allow` Directly?
+
+The common shortcut `ausearch -c grep --raw | audit2allow -M my-policy` can fail if:
+
+- The AVC events have already rotated out of the audit log
+- `ausearch` returns no matching records (outputs "Nothing to do")
+
+Writing the `.te` file manually is more reliable and self-documenting.
+
+## Environment
+
+- **OS:** Fedora 43
+- **SELinux:** Enforcing, targeted policy
+- **Fail2ban:** 1.1.0 (`fail2ban-selinux-1.1.0-15.fc43.noarch`)
+- **Kernel:** 6.19.x
+
+## See Also
+
+- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](../../05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — another SELinux fix for post-reboot service issues
+- [SELinux: Fixing Dovecot Mail Spool Context](../../05-troubleshooting/selinux-dovecot-vmail-context.md) — custom SELinux context for mail spool

02-selfhosting/security/ssh-hardening-ansible-fleet.md

@@ -0,0 +1,138 @@
+---
+title: "SSH Hardening Fleet-Wide with Ansible"
+domain: selfhosting
+category: security
+tags: [ssh, ansible, security, hardening, fleet]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+# SSH Hardening Fleet-Wide with Ansible
+
+## Overview
+
+Default SSH daemon settings on both Ubuntu and Fedora/RHEL are permissive. A drop-in configuration file (`/etc/ssh/sshd_config.d/99-hardening.conf`) lets you tighten settings without touching the distro-managed base config — and Ansible can deploy it atomically across every fleet host with a single playbook run.
+
+## Settings to Change
+
+| Setting | Default | Hardened | Reason |
+|---|---|---|---|
+| `PermitRootLogin` | `yes` | `without-password` | Prevent password-based root login; key auth still works for Ansible |
+| `X11Forwarding` | `yes` | `no` | Nothing in a typical homelab fleet uses X11 tunneling |
+| `AllowTcpForwarding` | `yes` | `no` | Eliminates a tunneling vector if a service account is compromised |
+| `MaxAuthTries` | `6` | `3` | Cuts per-connection brute-force attempts in half |
+| `LoginGraceTime` | `120` | `30` | Reduces the window for slow-connect attacks |
+
+## The Drop-in Approach
+
+Rather than editing `/etc/ssh/sshd_config` directly (which may be managed by the distro or overwritten on upgrades), place overrides in `/etc/ssh/sshd_config.d/99-hardening.conf`. The `Include /etc/ssh/sshd_config.d/*.conf` directive in the base config loads these in alphabetical order, and **first match wins** — so `99-` ensures your overrides come last and take precedence.
+
+> **Fedora/RHEL gotcha:** Fedora ships `/etc/ssh/sshd_config.d/50-redhat.conf` which sets `X11Forwarding yes`. Because first-match-wins applies, `50-redhat.conf` loads before `99-hardening.conf` and wins. You must patch `50-redhat.conf` in-place before deploying your drop-in, or the X11Forwarding setting will be silently ignored.
+
+## Ansible Playbook
+
+```yaml
+- name: Harden SSH daemon fleet-wide
+  hosts: all:!raspbian
+  become: true
+  gather_facts: true
+
+  tasks:
+
+    - name: Ensure sshd_config.d directory exists
+      ansible.builtin.file:
+        path: /etc/ssh/sshd_config.d
+        state: directory
+        owner: root
+        group: root
+        mode: '0755'
+
+    - name: Ensure Include directive is present in sshd_config
+      ansible.builtin.lineinfile:
+        path: /etc/ssh/sshd_config
+        line: "Include /etc/ssh/sshd_config.d/*.conf"
+        insertbefore: BOF
+        state: present
+
+    # Fedora only: neutralize 50-redhat.conf's X11Forwarding yes
+    # (first-match-wins means it would override our 99- drop-in)
+    - name: Comment out X11Forwarding in 50-redhat.conf (Fedora)
+      ansible.builtin.replace:
+        path: /etc/ssh/sshd_config.d/50-redhat.conf
+        regexp: '^(X11Forwarding yes)'
+        replace: '# \1  # disabled by ansible hardening'
+      when: ansible_os_family == "RedHat"
+      ignore_errors: true
+
+    - name: Deploy SSH hardening drop-in
+      ansible.builtin.copy:
+        dest: /etc/ssh/sshd_config.d/99-hardening.conf
+        content: |
+          # Managed by Ansible — do not edit manually
+          PermitRootLogin without-password
+          X11Forwarding no
+          AllowTcpForwarding no
+          MaxAuthTries 3
+          LoginGraceTime 30
+        owner: root
+        group: root
+        mode: '0644'
+      notify: Reload sshd
+
+    - name: Verify effective SSH settings
+      ansible.builtin.command:
+        cmd: sshd -T
+      register: sshd_effective
+      changed_when: false
+
+    - name: Assert hardened settings are active
+      ansible.builtin.assert:
+        that:
+          - "'permitrootlogin without-password' in sshd_effective.stdout"
+          - "'x11forwarding no' in sshd_effective.stdout"
+          - "'allowtcpforwarding no' in sshd_effective.stdout"
+          - "'maxauthtries 3' in sshd_effective.stdout"
+          - "'logingracetime 30' in sshd_effective.stdout"
+        fail_msg: "One or more SSH hardening settings not effective — check for conflicting config"
+      when: not ansible_check_mode
+
+  handlers:
+
+    - name: Reload sshd
+      ansible.builtin.service:
+        # Ubuntu/Debian: 'ssh' | Fedora/RHEL: 'sshd'
+        name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
+        state: reloaded
+```
+
+## Edge Cases
+
+**Ubuntu vs Fedora service name:** The SSH daemon is `ssh` on Debian/Ubuntu and `sshd` on Fedora/RHEL. The handler uses `ansible_os_family` to pick the right name automatically.
+
+**Missing Include directive:** Some minimal installs don't have `Include /etc/ssh/sshd_config.d/*.conf` in their base config. The `lineinfile` task adds it if absent. Without this, the drop-in directory exists but is never loaded.
+
+**Fedora's 50-redhat.conf:** Sets `X11Forwarding yes` with first-match priority. The playbook patches it before deploying the drop-in.
+
+**`sshd -T` in check mode:** `sshd -T` reads the *current* running config, not the pending changes. The assert task is guarded with `when: not ansible_check_mode` to prevent false failures during dry runs.
+
+**PermitRootLogin on hosts that already had it set:** Some hosts (e.g., those managed by another tool) may already have `PermitRootLogin without-password` set elsewhere. The drop-in still applies cleanly — it just becomes a no-op for that setting.
+
+## Verify Manually
+
+```bash
+# Check effective settings on any host
+ssh root@<host> "sshd -T | grep -E 'permitrootlogin|x11forwarding|allowtcpforwarding|maxauthtries|logingracetime'"
+
+# Expected:
+# permitrootlogin without-password
+# x11forwarding no
+# allowtcpforwarding no
+# maxauthtries 3
+# logingracetime 30
+```
+
+## See Also
+
+- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)
+- [ansible-unattended-upgrades-fleet](ansible-unattended-upgrades-fleet.md)
+- [ufw-firewall-management](ufw-firewall-management.md)

02-selfhosting/security/ufw-firewall-management.md

@@ -0,0 +1,192 @@
+---
+title: "UFW Firewall Management"
+domain: selfhosting
+category: security
+tags: [security, firewall, ufw, ubuntu, networking]
+status: published
+created: 2026-04-02
+updated: 2026-04-03
+---
+
+# UFW Firewall Management
+
+UFW (Uncomplicated Firewall) is the standard firewall tool on Ubuntu. It wraps iptables/nftables into something you can actually manage without losing your mind. This covers the syntax and patterns I use across the MajorsHouse fleet.
+
+## The Short Answer
+
+```bash
+# Enable UFW
+sudo ufw enable
+
+# Allow a port
+sudo ufw allow 80
+
+# Block a specific IP
+sudo ufw insert 1 deny from 203.0.113.50
+
+# Check status
+sudo ufw status numbered
+```
+
+## Basic Rules
+
+### Allow by Port
+
+```bash
+# Allow HTTP and HTTPS
+sudo ufw allow 80
+sudo ufw allow 443
+
+# Allow a port range
+sudo ufw allow 6000:6010/tcp
+
+# Allow a named application profile
+sudo ufw allow 'Apache Full'
+```
+
+### Allow by Interface
+
+Useful when you only want traffic on a specific network interface — this is how SSH is restricted to Tailscale across the fleet:
+
+```bash
+# Allow SSH only on the Tailscale interface
+sudo ufw allow in on tailscale0 to any port 22
+
+# Then deny SSH globally (evaluated after the allow above)
+sudo ufw deny 22
+```
+
+Rule order matters. UFW evaluates rules top to bottom and stops at the first match.
+
+### Allow by Source IP
+
+```bash
+# Allow a specific IP to access SSH
+sudo ufw allow from 100.86.14.126 to any port 22
+
+# Allow a subnet
+sudo ufw allow from 192.168.50.0/24 to any port 22
+```
+
+## Blocking IPs
+
+### Insert Rules at the Top
+
+When blocking IPs, use `insert 1` to place the deny rule at the top of the chain. Otherwise it may never be evaluated because an earlier ALLOW rule matches first.
+
+```bash
+# Block a single IP
+sudo ufw insert 1 deny from 203.0.113.50
+
+# Block a subnet
+sudo ufw insert 1 deny from 203.0.113.0/24
+
+# Block an IP from a specific port only
+sudo ufw insert 1 deny from 203.0.113.50 to any port 443
+```
+
+### Don't Accumulate Manual Blocks
+
+Manual `ufw deny` rules pile up fast. On one of my servers, I found **30,142 manual DENY rules** — a 3 MB rules file that every packet had to traverse. Use Fail2ban for automated blocking instead. It manages bans with expiry and doesn't pollute your UFW rules.
+
+If you inherit a server with thousands of manual blocks:
+
+```bash
+# Nuclear option — reset and re-add only the rules you need
+sudo ufw --force reset
+sudo ufw allow 'Apache Full'
+sudo ufw allow in on tailscale0 to any port 22
+sudo ufw deny 22
+sudo ufw enable
+```
+
+## Managing Rules
+
+### View Rules
+
+```bash
+# Simple view
+sudo ufw status
+
+# Numbered (needed for deletion and insert position)
+sudo ufw status numbered
+
+# Verbose (shows default policies and logging)
+sudo ufw status verbose
+```
+
+### Delete Rules
+
+```bash
+# Delete by rule number
+sudo ufw delete 3
+
+# Delete by rule specification
+sudo ufw delete allow 8080
+```
+
+### Default Policies
+
+```bash
+# Deny all incoming, allow all outgoing (recommended baseline)
+sudo ufw default deny incoming
+sudo ufw default allow outgoing
+```
+
+## Don't Forget Web Server Ports
+
+If you're running a web server behind UFW, make sure ports 80 and 443 are explicitly allowed. This sounds obvious, but it's easy to miss — especially on servers where UFW was enabled after the web server was already running, or where a firewall reset dropped rules that were never persisted.
+
+```bash
+# Allow HTTP and HTTPS
+sudo ufw allow 80
+sudo ufw allow 443
+
+# Or use an application profile
+sudo ufw allow 'Apache Full'
+```
+
+If your site suddenly stops responding after enabling UFW or resetting rules, check `sudo ufw status numbered` first. Missing web ports is the most common cause.
+
+## UFW with Fail2ban
+
+On Ubuntu servers, Fail2ban and UFW operate at different layers. Fail2ban typically creates its own nftables table (`inet f2b-table`) at a higher priority than UFW's chains. This means:
+
+- Fail2ban bans take effect **before** UFW rules are evaluated
+- A banned IP is rejected even if UFW has an ALLOW rule for that port
+- Add trusted IPs (your own, monitoring, etc.) to `ignoreip` in `/etc/fail2ban/jail.local` to prevent self-lockout
+
+```ini
+# /etc/fail2ban/jail.local
+[DEFAULT]
+ignoreip = 127.0.0.1/8 ::1 100.0.0.0/8
+```
+
+The `100.0.0.0/8` range covers all Tailscale IPs, which prevents banning fleet traffic.
+
+## UFW Logging
+
+```bash
+# Enable logging (low/medium/high/full)
+sudo ufw logging medium
+```
+
+Logs go to `/var/log/ufw.log`. Useful for seeing what's getting blocked, but `medium` or `low` is usually enough — `high` and `full` can be noisy.
+
+## Fleet Reference
+
+UFW is used on these MajorsHouse servers:
+
+| Host | Key UFW Rules |
+|---|---|
+| majortoot | SSH on tailscale0, deny 22 globally |
+| majorlinux | SSH on tailscale0, deny 22 globally |
+| tttpod | SSH on tailscale0, deny 22 globally, Apache Full (added 2026-04-03) |
+| teelia | SSH on tailscale0, deny 22 globally, Apache Full |
+
+The Fedora servers (majorlab, majorhome, majormail, majordiscord) use iptables or firewalld instead.
+
+## See Also
+
+- [Linux Server Hardening Checklist](linux-server-hardening-checklist.md) — initial firewall setup as part of server provisioning
+- [Fail2ban & UFW Rule Bloat Cleanup](../../05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md) — what happens when manual blocks get out of hand

02-selfhosting/services/mastodon-instance-tuning.md

@@ -0,0 +1,68 @@
+---
+title: "Mastodon Instance Tuning"
+domain: selfhosting
+category: services
+tags: [mastodon, fediverse, self-hosting, majortoot, docker]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+
+# Mastodon Instance Tuning
+
+Running your own Mastodon instance means you control the rules — including limits the upstream project imposes by default. These are the tweaks applied to **majortoot** (MajorsHouse's Mastodon instance).
+
+## Increase Character Limit
+
+Mastodon's default 500-character post limit is low for longer-form thoughts. You can raise it, but it requires modifying the source — there's no config toggle.
+
+The process depends on your deployment method (Docker vs bare metal) and Mastodon version. The community-maintained guide covers the approaches:
+
+- [How to increase the max number of characters of a post](https://qa.mastoadmin.social/questions/10010000000000011/how-do-i-increase-the-max-number-of-characters-of-a-post)
+
+**Key points:**
+- The limit is enforced in both the backend (Ruby) and frontend (React). Both must be changed or the UI will reject posts the API would accept.
+- After changing, you need to rebuild assets and restart services.
+- Other instances will still display the full post — the character limit is per-instance, not a federation constraint.
+- Some Mastodon forks (Glitch, Hometown) expose this as a config option without source patches.
+
+## Media Cache Management
+
+Federated content (avatars, headers, media from remote posts) gets cached locally. On a small instance this grows slowly, but over months it adds up — especially if you follow active accounts on large instances.
+
+Reference: [Fedicache — Understanding Mastodon's media cache](https://notes.neatnik.net/2024/08/fedicache)
+
+**Clean up cached remote media:**
+
+```bash
+# Preview what would be removed (older than 7 days)
+tootctl media remove --days 7 --dry-run
+
+# Actually remove it
+tootctl media remove --days 7
+
+# For Docker deployments
+docker exec mastodon-web tootctl media remove --days 7
+```
+
+**Automate with cron or systemd timer:**
+
+```bash
+# Weekly cache cleanup — crontab
+0 3 * * 0 docker exec mastodon-web tootctl media remove --days 7
+```
+
+**What gets removed:** Only cached copies of remote media. Local uploads (your posts, your users' posts) are never touched. Remote media will be re-fetched on demand if someone views the post again.
+
+**Storage impact:** On a single-user instance, remote media cache can still reach several GB over a few months of active federation. Regular cleanup keeps disk usage predictable.
+
+## Gotchas & Notes
+
+- **Character limit changes break on upgrades.** Any source patch gets overwritten when you pull a new Mastodon release. Track your changes and reapply after updates.
+- **`tootctl` is your admin CLI.** It handles media cleanup, user management, federation diagnostics, and more. Run `tootctl --help` for the full list.
+- **Monitor disk usage.** Even with cache cleanup, the PostgreSQL database and local media uploads grow over time. Keep an eye on it.
+
+## See Also
+
+- [self-hosting-starter-guide](../docker/self-hosting-starter-guide.md)
+- [docker-healthchecks](../docker/docker-healthchecks.md)

02-selfhosting/services/updating-n8n-docker.md

@@ -0,0 +1,121 @@
+---
+title: "Updating n8n Running in Docker"
+domain: selfhosting
+category: services
+tags: [n8n, docker, update, self-hosting, automation]
+status: published
+created: 2026-03-30
+updated: 2026-03-30
+---
+
+# Updating n8n Running in Docker
+
+n8n's in-app update notification checks against their npm release version, which often gets published before the `latest` Docker Hub tag is updated. This means you may see an update prompt in the UI even though `docker pull` reports the image as current. Pull a pinned version tag instead.
+
+## Check Current vs Latest Version
+
+```bash
+# Check what's running
+docker exec n8n-n8n-1 n8n --version
+
+# Check what npm (n8n's upstream) says is latest
+docker exec n8n-n8n-1 npm show n8n version
+```
+
+If the versions differ, the Docker Hub `latest` tag hasn't caught up yet. Use the pinned version tag.
+
+## Get the Running Container's Config
+
+Before stopping anything, capture the full environment so you can recreate the container identically:
+
+```bash
+docker inspect n8n-n8n-1 --format '{{json .Config.Env}}'
+docker inspect n8n-n8n-1 --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'
+```
+
+For MajorsHouse, the relevant env vars are:
+
+```
+N8N_EDITOR_BASE_URL=https://n8n.majorshouse.com/
+N8N_PORT=5678
+TZ=America/New_York
+N8N_TRUST_PROXY=true
+GENERIC_TIMEZONE=America/New_York
+N8N_HOST=n8n.majorshouse.com
+N8N_PROTOCOL=https
+WEBHOOK_URL=https://n8n.majorshouse.com/
+```
+
+Data volume: `n8n_n8n_data:/home/node/.n8n`
+
+## Perform the Update
+
+```bash
+# 1. Pull the specific version (replace 2.14.2 with target version)
+docker pull docker.n8n.io/n8nio/n8n:2.14.2
+
+# 2. Stop and remove the old container
+docker stop n8n-n8n-1 && docker rm n8n-n8n-1
+
+# 3. Start fresh with the new image and same settings
+docker run -d \
+  --name n8n-n8n-1 \
+  --restart unless-stopped \
+  -p 127.0.0.1:5678:5678 \
+  -v n8n_n8n_data:/home/node/.n8n \
+  -e N8N_EDITOR_BASE_URL=https://n8n.majorshouse.com/ \
+  -e N8N_PORT=5678 \
+  -e TZ=America/New_York \
+  -e N8N_TRUST_PROXY=true \
+  -e GENERIC_TIMEZONE=America/New_York \
+  -e N8N_HOST=n8n.majorshouse.com \
+  -e N8N_PROTOCOL=https \
+  -e WEBHOOK_URL=https://n8n.majorshouse.com/ \
+  docker.n8n.io/n8nio/n8n:2.14.2
+
+# 4. Verify
+docker exec n8n-n8n-1 n8n --version
+docker ps --filter name=n8n-n8n-1 --format '{{.Status}}'
+```
+
+No restart of Caddy or other services required. Workflows, credentials, and execution history are preserved in the data volume.
+
+## Reset a Forgotten Admin Password
+
+n8n uses SQLite at `/home/node/.n8n/database.sqlite` (mapped to `n8n_n8n_data` on the host). Use Python to generate a valid bcrypt hash and update it directly — do **not** use shell variable interpolation, as `$` characters in bcrypt hashes will be eaten.
+
+```bash
+python3 -c "
+import bcrypt, sqlite3
+pw = b'your-new-password'
+h = bcrypt.hashpw(pw, bcrypt.gensalt(rounds=10)).decode()
+db = sqlite3.connect('/var/lib/docker/volumes/n8n_n8n_data/_data/database.sqlite')
+db.execute(\"UPDATE user SET password=? WHERE email='marcus@majorshouse.com'\", (h,))
+db.commit()
+db.close()
+db2 = sqlite3.connect('/var/lib/docker/volumes/n8n_n8n_data/_data/database.sqlite')
+row = db2.execute(\"SELECT password FROM user WHERE email='marcus@majorshouse.com'\").fetchone()
+print('Valid:', bcrypt.checkpw(pw, row[0].encode()))
+"
+```
+
+`Valid: True` confirms the hash is correct. No container restart needed.
+
+## Why Arcane Doesn't Always Catch It
+
+[Arcane](https://getarcaneapp.com) watches Docker Hub for image digest changes. When n8n publishes a new release, there's often a delay before the `latest` tag on Docker Hub is updated to match. During that window:
+
+- n8n's in-app updater (checks npm) reports an update available
+- `docker pull latest` and Arcane both report the image as current
+
+Once Docker Hub catches up, Arcane will notify normally. For immediate updates, use pinned version tags as shown above.
+
+## Troubleshooting
+
+**Password still rejected after update:** Shell variable interpolation (`$2b`, `$10`, etc.) silently truncates bcrypt hashes when passed as inline SQL strings. Always use the Python script approach above.
+
+**Container exits immediately after recreate:** Check `docker logs n8n-n8n-1`. Most commonly a missing env var or a volume permission issue.
+
+**Webhooks not firing after update:** Verify `N8N_TRUST_PROXY=true` is set. Without it, Caddy's `X-Forwarded-For` header causes n8n's rate limiter to drop webhook requests before parsing the body.
+
+**`npm show n8n version` returns old version:** npm registry cache inside the container. Run `docker exec n8n-n8n-1 npm show n8n version --no-cache` to force a fresh check.

02-selfhosting/storage-backup/rsync-backup-patterns.md

@@ -148,6 +148,29 @@ WantedBy=timers.target
 sudo systemctl enable --now rsync-backup.timer
 ```
 
+## Cold Storage — AWS Glacier Deep Archive
+
+rsync handles local and remote backups, but for true offsite cold storage — disaster recovery, archival copies you rarely need to retrieve — AWS Glacier Deep Archive is the cheapest option at ~$1/TB/month.
+
+Upload files directly to an S3 bucket with the `DEEP_ARCHIVE` storage class:
+
+```bash
+# Single file
+aws s3 cp backup.tar.gz s3://your-bucket/ --storage-class DEEP_ARCHIVE
+
+# Entire directory
+aws s3 sync /backup/offsite/ s3://your-bucket/offsite/ --storage-class DEEP_ARCHIVE
+```
+
+**When to use it:** Long-term backups you'd only need in a disaster scenario — media archives, yearly snapshots, irreplaceable data. Not for anything you'd need to restore quickly.
+
+**Retrieval tradeoffs:**
+- **Standard retrieval:** 12 hours, cheapest restore cost
+- **Bulk retrieval:** Up to 48 hours, even cheaper
+- **Expedited:** Not available for Deep Archive — if you need faster access, use regular Glacier or S3 Infrequent Access
+
+**In the MajorsHouse backup strategy**, rsync handles the daily local and cross-host backups. Glacier Deep Archive is the final tier — offsite, durable, cheap, and slow to retrieve by design. A good backup plan has both.
+
 ## Gotchas & Notes
 
 - **Test with `--dry-run` first.** Especially when using `--delete`. See what would be removed before actually removing it.
@@ -158,5 +181,5 @@ sudo systemctl enable --now rsync-backup.timer
 
 ## See Also
 
-- [[self-hosting-starter-guide]]
+- [self-hosting-starter-guide](../docker/self-hosting-starter-guide.md)
-- [[bash-scripting-patterns]]
+- [bash-scripting-patterns](../../01-linux/shell-scripting/bash-scripting-patterns.md)

03-opensource/alternatives/freshrss.md

@@ -1,3 +1,12 @@
+---
+title: "FreshRSS — Self-Hosted RSS Reader"
+domain: opensource
+category: alternatives
+tags: [freshrss, rss, self-hosting, docker, privacy]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # FreshRSS — Self-Hosted RSS Reader
 
 ## Problem
@@ -83,7 +92,3 @@ The `CRON_MIN=*/15` environment variable runs feed fetching every 15 minutes ins
 - **Works forever** — RSS has been around since 1999 and isn't going anywhere
 
 ---
-
-## Tags
-
-#freshrss #rss #self-hosting #docker #linux #alternatives #privacy

03-opensource/alternatives/gitea.md

@@ -1,3 +1,12 @@
+---
+title: "Gitea — Self-Hosted Git"
+domain: opensource
+category: alternatives
+tags: [gitea, git, self-hosting, docker, ci-cd]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Gitea — Self-Hosted Git
 
 ## Problem
@@ -89,7 +98,3 @@ For public open source — GitHub is fine, the network effects are real. For pri
 - Works entirely over Tailscale — no public exposure required
 
 ---
-
-## Tags
-
-#gitea #git #self-hosting #docker #linux #alternatives #vcs

03-opensource/alternatives/searxng.md

@@ -1,3 +1,12 @@
+---
+title: "SearXNG — Private Self-Hosted Search"
+domain: opensource
+category: alternatives
+tags: [searxng, search, privacy, self-hosting, docker]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # SearXNG — Private Self-Hosted Search
 
 ## Problem
@@ -82,7 +91,3 @@ DDG is better than Google for privacy, but it's still a centralized third-party
 - Can be kept entirely off the public internet (Tailscale-only)
 
 ---
-
-## Tags
-
-#searxng #search #privacy #self-hosting #docker #linux #alternatives

03-opensource/dev-tools/rsync.md

@@ -1,3 +1,12 @@
+---
+title: "rsync — Fast, Resumable File Transfers"
+domain: opensource
+category: dev-tools
+tags: [rsync, backup, file-transfer, linux, cli]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # rsync — Fast, Resumable File Transfers
 
 ## Problem
@@ -96,7 +105,3 @@ tail -f /root/raid_migrate.log
 ```
 
 ---
-
-## Tags
-
-#rsync #linux #storage #file-transfer #sysadmin #dev-tools

03-opensource/dev-tools/screen.md

@@ -1,3 +1,12 @@
+---
+title: "screen — Simple Persistent Terminal Sessions"
+domain: opensource
+category: dev-tools
+tags: [screen, terminal, ssh, linux, cli]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # screen — Simple Persistent Terminal Sessions
 
 ## Problem
@@ -70,7 +79,3 @@ screen -S mysession -X stuff "tail -f /root/output.log\n"
 Use screen when it's already there or for quick throwaway sessions. Use tmux for anything more complex. See [tmux](tmux.md).
 
 ---
-
-## Tags
-
-#screen #terminal #linux #ssh #productivity #dev-tools

03-opensource/dev-tools/tmux.md

@@ -1,3 +1,12 @@
+---
+title: "tmux — Persistent Terminal Sessions"
+domain: opensource
+category: dev-tools
+tags: [tmux, terminal, ssh, multiplexer, linux]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # tmux — Persistent Terminal Sessions
 
 ## Problem
@@ -87,7 +96,3 @@ tmux ls  # check what's running
 Both work. tmux has better split-pane support and scripting. screen is simpler and more universally installed. I use both — tmux for new jobs, screen for legacy ones. See the [screen](screen.md) article for reference.
 
 ---
-
-## Tags
-
-#tmux #terminal #linux #ssh #productivity #dev-tools

03-opensource/dev-tools/ventoy.md

@@ -0,0 +1,81 @@
+---
+title: "Ventoy — Multi-Boot USB Tool"
+domain: opensource
+category: dev-tools
+tags: [ventoy, usb, boot, iso, linux, tools]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+
+# Ventoy — Multi-Boot USB Tool
+
+Ventoy turns a USB drive into a multi-boot device. Drop ISO files onto the drive and boot directly from them — no need to flash a new image every time you want to try a different distro or run a recovery tool.
+
+## What It Is
+
+[Ventoy](https://www.ventoy.net/) creates a special partition layout on a USB drive. After the one-time install, you just copy ISO (or WIM, VHD, IMG) files to the drive. On boot, Ventoy presents a menu of every image on the drive and boots whichever one you pick.
+
+No re-formatting. No Rufus. No balenaEtcher. Just drag and drop.
+
+## Installation
+
+### Linux
+
+```bash
+# Download the latest release
+wget https://github.com/ventoy/Ventoy/releases/download/v1.1.05/ventoy-1.1.05-linux.tar.gz
+
+# Extract
+tar -xzf ventoy-1.1.05-linux.tar.gz
+cd ventoy-1.1.05
+
+# Install to USB drive (WARNING: this formats the drive)
+sudo ./Ventoy2Disk.sh -i /dev/sdX
+```
+
+Replace `/dev/sdX` with your USB drive. Use `lsblk` to identify it — triple-check before running, this wipes the drive.
+
+### Windows
+
+Download the Windows package from the Ventoy releases page, run `Ventoy2Disk.exe`, select your USB drive, and click Install.
+
+## Usage
+
+After installation, the USB drive shows up as a regular FAT32/exFAT partition. Copy ISOs onto it:
+
+```bash
+# Copy ISOs to the drive
+cp ~/Downloads/Fedora-43-x86_64.iso /mnt/ventoy/
+cp ~/Downloads/ubuntu-24.04-desktop.iso /mnt/ventoy/
+cp ~/Downloads/memtest86.iso /mnt/ventoy/
+```
+
+Boot from the USB. Ventoy's menu lists every ISO it finds. Select one and it boots directly.
+
+## Updating Ventoy
+
+When a new version comes out, update without losing your ISOs:
+
+```bash
+# Update mode (-u) preserves existing files
+sudo ./Ventoy2Disk.sh -u /dev/sdX
+```
+
+## Why It's Useful
+
+- **Distro testing:** Keep 5-10 distro ISOs on one stick. Boot into any of them without reflashing.
+- **Recovery toolkit:** Carry GParted, Clonezilla, memtest86, and a live Linux on a single drive.
+- **OS installation:** One USB for every machine you need to set up.
+- **Persistence:** Ventoy supports persistent storage for some distros, so live sessions can save data across reboots.
+
+## Gotchas & Notes
+
+- **Secure Boot:** Ventoy supports Secure Boot but it requires enrolling a key on first boot. Follow the on-screen prompts.
+- **exFAT for large ISOs:** The default FAT32 partition has a 4GB file size limit. Use exFAT if any of your ISOs exceed that (Windows ISOs often do). Ventoy supports both.
+- **UEFI vs Legacy:** Ventoy handles both automatically. It detects the boot mode and presents the appropriate menu.
+- **Some ISOs don't work.** Heavily customized or non-standard ISOs may fail to boot. Standard distro ISOs and common tools work reliably.
+
+## See Also
+
+- [linux-distro-guide-beginners](../../01-linux/distro-specific/linux-distro-guide-beginners.md)

03-opensource/media-creative/yt-dlp.md

@@ -1,3 +1,12 @@
+---
+title: "yt-dlp — Video Downloading"
+domain: opensource
+category: media-creative
+tags: [yt-dlp, video, youtube, downloads, cli]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # yt-dlp — Video Downloading
 
 ## What It Is
@@ -118,12 +127,31 @@ tail -f ~/yt-download.log
 
 ---
 
+## Subtitle Downloads
+
+The config above handles subtitles automatically via `--write-auto-subs` and `--embed-subs`. For one-off downloads where you want explicit control over subtitle embedding alongside specific format selection:
+
+```bash
+yt-dlp -f 'bestvideo[vcodec^=avc]+bestaudio[ext=m4a]/bestvideo+bestaudio' \
+    --merge-output-format mp4 \
+    -o "/plex/plex/%(title)s.%(ext)s" \
+    --write-auto-subs --embed-subs URL
+```
+
+This forces H.264 video + M4A audio when available — useful when you want guaranteed Apple TV / Plex compatibility without running the HEVC conversion hook.
+
+---
+
 ## Troubleshooting
 
 For YouTube JS challenge errors, missing formats, and n-challenge failures on Fedora — see [yt-dlp YouTube JS Challenge Fix](../../05-troubleshooting/yt-dlp-fedora-js-challenge.md).
 
+**YouTube player client errors:** If downloads fail with extractor errors, YouTube may have broken the default player client. Override it:
+
+```bash
+yt-dlp --extractor-args "youtube:player-client=default,-tv_simply" URL
+```
+
+This can also be added to your config file as a persistent workaround until yt-dlp pushes a fix upstream. Keep yt-dlp updated — these breakages get patched regularly.
+
 ---
-
-## Tags
-
-#yt-dlp #youtube #video #plex #linux #media #dev-tools

03-opensource/privacy-security/vaultwarden.md

@@ -1,3 +1,12 @@
+---
+title: "Vaultwarden — Self-Hosted Password Manager"
+domain: opensource
+category: privacy-security
+tags: [vaultwarden, bitwarden, passwords, self-hosting, docker]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Vaultwarden — Self-Hosted Password Manager
 
 ## Problem
@@ -89,7 +98,3 @@ Or include the `vw-data/` directory in your regular rsync backup run.
 The official Bitwarden server is also open source but requires significantly more resources (multiple services, SQL Server). Vaultwarden runs in a single container on minimal RAM and handles everything a personal or family vault needs.
 
 ---
-
-## Tags
-
-#vaultwarden #bitwarden #passwords #privacy #self-hosting #docker #linux

03-opensource/productivity/rmlint-duplicate-scanning.md

@@ -1,3 +1,12 @@
+---
+title: "rmlint — Extreme Duplicate File Scanning"
+domain: opensource
+category: productivity
+tags: [rmlint, duplicates, storage, cleanup, linux]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # rmlint — Extreme Duplicate File Scanning
 
 ## Problem
@@ -52,7 +61,3 @@ After scanning and clearing duplicates, you can reclaim significant space. In my
 Run a scan monthly or before any major storage consolidation project.
 
 ---
-
-## Tags
-
-#rmlint #linux #storage #cleanup #duplicates

04-streaming/obs/obs-studio-setup-encoding.md

@@ -118,6 +118,31 @@ echo "v4l2loopback" | sudo tee /etc/modules-load.d/v4l2loopback.conf
 echo "options v4l2loopback devices=1 video_nr=10 card_label=OBS Virtual Camera exclusive_caps=1" | sudo tee /etc/modprobe.d/v4l2loopback.conf
 ```
 
+## Plugins & Capture Sources
+
+### Captions Plugin (Accessibility)
+
+[OBS Captions Plugin](https://github.com/ratwithacompiler/OBS-captions-plugin) adds real-time closed captions to streams using speech-to-text. Viewers can toggle captions on/off in their player — important for accessibility and for viewers watching without sound.
+
+Install from the plugin's GitHub releases page, then configure in Tools → Captions.
+
+### VLC Video Source (Capture Card)
+
+For capturing from an Elgato 4K60 Pro MK.2 (or similar DirectShow capture card) via VLC as an OBS source, use this device string:
+
+```
+:dshow-vdev=Game Capture 4K60 Pro MK.2
+:dshow-adev=Game Capture 4K60 Pro MK.2 Audio (Game Capture 4K60 Pro MK.2)
+:dshow-aspect-ratio=16:9
+:dshow-chroma=YUY2
+:dshow-fps=0
+:no-dshow-config
+:no-dshow-tuner
+:live-caching=0
+```
+
+Set `live-caching=0` to minimize capture latency. This is useful when OBS's native Game Capture isn't an option (e.g., capturing a separate machine's output through the card).
+
 ## Gotchas & Notes
 
 - **Test your stream before going live.** Record a short clip and watch it back. Artifacts in the recording will be worse in the stream.
@@ -128,5 +153,5 @@ echo "options v4l2loopback devices=1 video_nr=10 card_label=OBS Virtual Camera e
 
 ## See Also
 
-- [[linux-file-permissions]]
+- [linux-file-permissions](../../01-linux/files-permissions/linux-file-permissions.md)
-- [[bash-scripting-patterns]]
+- [bash-scripting-patterns](../../01-linux/shell-scripting/bash-scripting-patterns.md)

04-streaming/plex/plex-4k-codec-compatibility.md

@@ -1,3 +1,12 @@
+---
+title: "Plex 4K Codec Compatibility (Apple TV)"
+domain: streaming
+category: plex
+tags: [plex, 4k, hevc, apple-tv, transcoding, codec]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Plex 4K Codec Compatibility (Apple TV)
 
 4K content on YouTube is delivered in AV1 or VP9 — neither of which the Plex app on Apple TV can direct play. This forces Plex to transcode, and most home server CPUs can't transcode 4K in real time. The fix is converting to HEVC before Plex ever sees the file.

05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md

@@ -0,0 +1,72 @@
+---
+title: Ansible SSH Timeout During dnf upgrade on Fedora Hosts
+domain: troubleshooting
+category: ansible
+tags:
+  - ansible
+  - ssh
+  - fedora
+  - dnf
+  - timeout
+  - fleet-management
+status: published
+created: '2026-03-28'
+updated: '2026-03-28'
+---
+
+# Ansible SSH Timeout During dnf upgrade on Fedora Hosts
+
+## Symptom
+
+Running `ansible-playbook update.yml` against Fedora/CentOS hosts fails with:
+
+```
+fatal: [hostname]: UNREACHABLE! => {"changed": false,
+  "msg": "Failed to connect to the host via ssh: Shared connection to <IP> closed."}
+```
+
+The failure occurs specifically during `ansible.builtin.dnf` tasks that upgrade all packages (`name: '*'`, `state: latest`), because the operation takes long enough for the SSH connection to drop.
+
+## Root Cause
+
+Without explicit SSH keepalive settings in `ansible.cfg`, OpenSSH defaults apply. Long-running tasks like full `dnf upgrade` across a fleet can exceed idle timeouts, causing the control connection to close mid-task.
+
+## Fix
+
+Add a `[ssh_connection]` section to `ansible.cfg`:
+
+```ini
+[ssh_connection]
+ssh_args = -o ServerAliveInterval=30 -o ServerAliveCountMax=10 -o ControlMaster=auto -o ControlPersist=60s
+```
+
+| Setting | Purpose |
+|---------|---------|
+| `ServerAliveInterval=30` | Send a keepalive every 30 seconds |
+| `ServerAliveCountMax=10` | Allow 10 missed keepalives before disconnect (~5 min tolerance) |
+| `ControlMaster=auto` | Reuse SSH connections across tasks |
+| `ControlPersist=60s` | Keep the master connection open 60s after last use |
+
+## Related Fix: do-agent Task Guard
+
+In the same playbook run, a second failure surfaced on hosts where the `ansible.builtin.uri` task to fetch the latest `do-agent` release was **skipped** (non-RedHat hosts or hosts without do-agent installed). The registered variable existed but contained a skipped result with no `.json` attribute, causing:
+
+```
+object of type 'dict' has no attribute 'json'
+```
+
+Fix: add guards to downstream tasks that reference the URI result:
+
+```yaml
+when:
+  - do_agent_release is defined
+  - do_agent_release is not skipped
+  - do_agent_release.json is defined
+```
+
+## Environment
+
+- **Controller:** macOS (MajorAir)
+- **Targets:** Fedora 43 (majorlab, majormail, majorhome, majordiscord)
+- **Ansible:** community edition via Homebrew
+- **Committed:** `d9c6bdb` in MajorAnsible repo

05-troubleshooting/ansible-vault-password-file-missing.md

@@ -1,3 +1,12 @@
+---
+title: "Ansible: Vault Password File Not Found"
+domain: troubleshooting
+category: general
+tags: [ansible, vault, credentials, configuration]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Ansible: Vault Password File Not Found
 
 ## Error

05-troubleshooting/ansible-wsl2-world-writable-mount-ignores-cfg.md

@@ -0,0 +1,89 @@
+---
+title: "Ansible Ignores ansible.cfg on WSL2 Windows Mounts"
+domain: troubleshooting
+category: ansible
+tags: [ansible, wsl, wsl2, windows, vault, configuration]
+status: published
+created: 2026-04-03
+updated: 2026-04-03
+---
+
+# Ansible Ignores ansible.cfg on WSL2 Windows Mounts
+
+## Problem
+
+Running Ansible from a repo on a Windows drive (`/mnt/c/`, `/mnt/d/`, etc.) in WSL2 silently ignores the local `ansible.cfg`. You'll see:
+
+```
+[WARNING]: Ansible is being run in a world writable directory
+(/mnt/d/MajorAnsible), ignoring it as an ansible.cfg source.
+```
+
+This causes vault decryption to fail (`Attempting to decrypt but no vault secrets found`), inventory to fall back to `/etc/ansible/hosts`, and `remote_user` to reset to defaults — even though `ansible.cfg` is right there in the project directory.
+
+## Cause
+
+WSL2 mounts Windows NTFS drives with broad permissions (typically `0777`). Ansible refuses to load `ansible.cfg` from any world-writable directory as a security measure — a malicious user on a shared system could inject a rogue config.
+
+This is hardcoded behavior in Ansible and cannot be overridden with a flag.
+
+## Solutions
+
+### Option 1: Environment Variables (Recommended)
+
+Export the settings that `ansible.cfg` would normally provide. Add to `~/.bashrc`:
+
+```bash
+export ANSIBLE_VAULT_PASSWORD_FILE=~/.ansible/vault_pass
+```
+
+Other common settings you may need:
+
+```bash
+export ANSIBLE_REMOTE_USER=root
+export ANSIBLE_INVENTORY=/mnt/d/MajorAnsible/inventory/inventory.yml
+```
+
+### Option 2: Pass Flags Explicitly
+
+```bash
+ansible-playbook -i inventory/ playbook.yml --vault-password-file ~/.ansible/vault_pass
+```
+
+This works but is tedious for daily use.
+
+### Option 3: Clone to a Native Linux Path
+
+Clone the repo inside the WSL2 filesystem instead of on the Windows mount:
+
+```bash
+git clone https://git.example.com/repo.git ~/MajorAnsible
+```
+
+Native WSL2 paths (`/home/user/...`) have proper Linux permissions, so `ansible.cfg` loads normally. The tradeoff is that Windows tools can't easily access the repo.
+
+### Option 4: Fix Mount Permissions (Not Recommended)
+
+You can change WSL2 mount permissions via `/etc/wsl.conf`:
+
+```ini
+[automount]
+options = "metadata,umask=022"
+```
+
+This requires a `wsl --shutdown` and remount. It may break other Windows-Linux interop workflows and affects all mounted drives.
+
+## Diagnosis
+
+To confirm whether Ansible is loading your config:
+
+```bash
+ansible --version
+```
+
+Look for the `config file` line. If it shows `None` instead of your project's `ansible.cfg`, the config is being ignored.
+
+## Related
+
+- [Ansible: Vault Password File Not Found](ansible-vault-password-file-missing.md) — general vault password troubleshooting
+- [Ansible Docs: Avoiding Security Risks with ansible.cfg](https://docs.ansible.com/ansible/latest/reference_appendices/config.html#cfg-in-world-writable-dir)

05-troubleshooting/claude-mem-setting-sources-empty-arg.md

@@ -0,0 +1,178 @@
+---
+title: "claude-mem Silently Fails with Claude Code 2.1+ (Empty --setting-sources)"
+domain: troubleshooting
+category: claude-code
+tags: [claude-code, claude-mem, cli, subprocess, version-mismatch, shim]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+
+# claude-mem Silently Fails with Claude Code 2.1+ (Empty `--setting-sources`)
+
+## Symptom
+
+After installing the `claude-mem` plugin (v12.1.3) in Claude Code (v2.1.112), every Claude Code session starts with:
+
+```
+No previous sessions found for this project yet.
+```
+
+…even for directories where you've worked repeatedly. Session records *do* appear in `~/.claude-mem/claude-mem.db` (table `sdk_sessions`), but:
+
+- `session_summaries` count stays at **0**
+- `observations` count stays at **0**
+- Chroma vector DB stays empty
+
+Tailing `~/.claude-mem/logs/claude-mem-YYYY-MM-DD.log` shows the Stop hook firing on every assistant turn, but always:
+
+```
+[HOOK  ] → Stop: Requesting summary {hasLastAssistantMessage=true}
+[HOOK  ] Summary processing complete {waitedMs=503, summaryStored=null}
+```
+
+No errors, no stack traces — just a silent `null`. Raising `CLAUDE_MEM_LOG_LEVEL` to `DEBUG` reveals the true error:
+
+```
+[WARN ] [SDK_SPAWN] Claude process exited {code=1, signal=null, pid=…}
+[ERROR] [SESSION] Generator failed {provider=claude, error=Claude Code process exited with code 1}
+```
+
+## Root cause
+
+`claude-mem` 12.1.3 spawns the `claude` CLI as a subprocess to generate per-turn observations and session summaries. The argv it passes includes:
+
+```
+claude --output-format stream-json --verbose --input-format stream-json \
+       --model claude-sonnet-4-6 \
+       --disallowedTools Bash,Read,Write,… \
+       --setting-sources \               ← no value!
+       --permission-mode default
+```
+
+`claude-mem` intends to pass `--setting-sources ""` (empty string, meaning "no sources"). Claude Code **v2.1.x** now validates this flag and rejects empty values — it requires one of `user`, `project`, or `local`. With no value present, the CLI's argument parser consumes the next flag (`--permission-mode`) as the value and produces:
+
+```
+Error processing --setting-sources: Invalid setting source: --permission-mode.
+Valid options are: user, project, local
+```
+
+The child process exits immediately with code 1 (within ~130 ms). `claude-mem` only logs `exited with code 1` and discards stderr by default, which is why the failure looks silent.
+
+This is a **version-mismatch bug** between `claude-mem` 12.1.3 (latest as of 2026-04-17) and `claude-code` 2.1.x. Earlier Claude Code releases accepted empty values.
+
+## Investigation path
+
+1. Confirm worker processes are alive:
+   ```bash
+   pgrep -fl "worker-service|mcp-server.cjs|chroma-mcp"
+   cat ~/.claude-mem/supervisor.json
+   ```
+2. Confirm sessions are being *recorded* but not *summarised*:
+   ```bash
+   sqlite3 ~/.claude-mem/claude-mem.db \
+     "SELECT COUNT(*) FROM sdk_sessions;  -- nonzero
+      SELECT COUNT(*) FROM session_summaries;  -- 0 = pipeline broken
+      SELECT COUNT(*) FROM observations;       -- 0 = pipeline broken"
+   ```
+3. Grep the log for `summaryStored=null` — if every Stop hook ends in `null`, summarisation is failing.
+4. Raise log level to expose the real error:
+   ```bash
+   # In ~/.claude-mem/settings.json
+   "CLAUDE_MEM_LOG_LEVEL": "DEBUG"
+   ```
+   Kill and respawn workers (`pkill -f worker-service.cjs`). New logs should show `SDK_SPAWN Claude process exited {code=1}`.
+5. Capture the exact argv by replacing `CLAUDE_CODE_PATH` with a debug shim that logs `$@` before exec'ing the real binary (see the fix below for the production shim — the debug version just tees argv to a log file).
+
+## The fix
+
+Apply in this order.
+
+### 1. Fix the settings `claude-mem` ships with empty
+
+Edit `~/.claude-mem/settings.json`:
+
+```json
+{
+  "CLAUDE_CODE_PATH": "/Users/you/.local/bin/claude-shim",
+  "CLAUDE_MEM_TIER_SUMMARY_MODEL": "claude-sonnet-4-6"
+}
+```
+
+Both ship empty in a fresh install. `CLAUDE_CODE_PATH` points at the shim (below), not the real binary. `CLAUDE_MEM_TIER_SUMMARY_MODEL` is required when `CLAUDE_MEM_TIER_ROUTING_ENABLED=true`.
+
+### 2. Install the shim
+
+`/Users/you/.local/bin/claude-shim`:
+
+```bash
+#!/bin/bash
+# Workaround shim for claude-mem 12.1.3 <-> Claude Code 2.1.x incompat.
+# claude-mem passes `--setting-sources` with no value; Claude CLI 2.1+ rejects
+# empty and consumes the next flag as the value. Fix: inject "user" when missing.
+
+REAL=/Users/you/.local/bin/claude
+
+new_args=()
+i=0
+args=("$@")
+while [ $i -lt ${#args[@]} ]; do
+  cur="${args[$i]}"
+  new_args+=("$cur")
+  if [ "$cur" = "--setting-sources" ]; then
+    next="${args[$((i+1))]}"
+    case "$next" in
+      user|project|local) : ;;   # already valid
+      *) new_args+=("user") ;;   # inject missing value
+    esac
+  fi
+  i=$((i+1))
+done
+
+exec "$REAL" "${new_args[@]}"
+```
+
+Chmod it executable: `chmod +x ~/.local/bin/claude-shim`.
+
+### 3. Restart workers
+
+```bash
+pkill -f "worker-service.cjs --daemon"
+```
+
+They respawn automatically on the next Claude Code hook fire. Verify:
+
+```bash
+# Within ~15 s:
+sqlite3 ~/.claude-mem/claude-mem.db "SELECT COUNT(*) FROM observations;"
+# Should be growing as you continue the session.
+```
+
+### 4. Sanity-check the shim is being used
+
+```bash
+ps -eww | grep -F 'setting-sources user'
+```
+
+Every live `claude` child should have `--setting-sources user` in its argv, not a bare `--setting-sources`.
+
+## Why a shim instead of patching `claude-mem`
+
+The offending code is inside the minified `worker-service.cjs` bundle shipped by `@anthropic-ai/claude-code` SDK, which `claude-mem` vendors. Patching the bundle is possible but fragile: any `claude-mem` update overwrites it. The shim is a one-file wrapper at a stable path, survives plugin updates, and becomes a no-op the moment upstream ships a fix.
+
+## When to remove the shim
+
+Check for a newer `claude-mem` release or an Anthropic SDK update that stops passing `--setting-sources` with an empty value. Test by:
+
+1. Point `CLAUDE_CODE_PATH` back at the real `/Users/you/.local/bin/claude`.
+2. Restart workers.
+3. Confirm `observations` count keeps growing.
+
+If it does, remove the shim. If not, restore the shim path and wait for a later release.
+
+## Related
+
+- Install notes: `20-Projects/Personal-Tasks.md` — "Install claude-mem plugin on MajorMac — 2026-04-15"
+- Config file: `~/.claude-mem/settings.json`
+- Logs: `~/.claude-mem/logs/claude-mem-YYYY-MM-DD.log`
+- DB: `~/.claude-mem/claude-mem.db` (SQLite, FTS5 enabled)

05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md

@@ -0,0 +1,84 @@
+---
+title: "Cron Heartbeat False Alarm: /var/run Cleared by Reboot"
+domain: troubleshooting
+category: general
+tags:
+  - cron
+  - systemd
+  - tmpfs
+  - monitoring
+  - backups
+  - heartbeat
+status: published
+created: 2026-04-13
+updated: 2026-04-13T10:10
+---
+# Cron Heartbeat False Alarm: /var/run Cleared by Reboot
+
+If a cron-driven watchdog emails you that a job "may never have run" — but the job's log clearly shows it completed successfully — check whether the heartbeat file lives under `/var/run` (or `/run`). On most modern Linux distros, `/run` is a **tmpfs** and is wiped on every reboot. Any file there survives only until the next boot.
+
+## Symptoms
+
+- A heartbeat-based watchdog fires a missing-heartbeat or stale-heartbeat alert
+- The job the watchdog is monitoring actually ran successfully — its log file shows a clean completion long before the alert fired
+- The host was rebooted between when the job wrote its heartbeat and when the watchdog checked it
+- `stat /var/run/<your-heartbeat>` returns `No such file or directory`
+- `readlink -f /var/run` returns `/run`, and `mount | grep ' /run '` shows `tmpfs`
+
+## Why It Happens
+
+Systemd distros mount `/run` as a tmpfs for runtime state. `/var/run` is kept only as a compatibility symlink to `/run`. The whole filesystem is memory-backed: when the host reboots, every file under `/run` vanishes unless a `tmpfiles.d` rule explicitly recreates it. The convention is that only things like PID files and sockets — state that is meaningful **only for the current boot** — should live there.
+
+A daily backup or maintenance job that touches a heartbeat file to prove it ran is *not* boot-scoped state. If the job runs at 03:00, the host reboots at 07:00 for a kernel update, and a watchdog checks the heartbeat at 08:00, the watchdog sees nothing — even though the job ran four hours earlier and exited 0.
+
+The common mitigation of checking the heartbeat's mtime against a max age (e.g. "alert if older than 25h") does **not** protect against this. It catches stale heartbeats from real failures, but a deleted file has no mtime to compare.
+
+## Fix
+
+Move the heartbeat out of tmpfs and into a persistent directory. Good options:
+
+- `/var/lib/<service>/heartbeat` — canonical home for persistent service state
+- `/var/log/<service>-heartbeat` — acceptable if you want it alongside existing logs
+- Any path on a real disk-backed filesystem
+
+Both the writer (the monitored job) and the reader (the watchdog) need to agree on the new path. Make sure the parent directory exists before the first write:
+
+```bash
+HEARTBEAT="/var/lib/myservice/heartbeat"
+mkdir -p "$(dirname "$HEARTBEAT")"
+# ... later, on success:
+touch "$HEARTBEAT"
+```
+
+The `mkdir -p` is cheap to run unconditionally and avoids a first-run-after-deploy edge case where the directory hasn't been created yet.
+
+## Verification
+
+After deploying the fix:
+
+```bash
+# 1. Run the monitored job manually (or wait for its next scheduled run)
+sudo bash /path/to/monitored-job.sh
+
+# 2. Confirm the heartbeat was created on persistent storage
+ls -la /var/lib/myservice/heartbeat
+
+# 3. Reboot and re-check — the file should survive
+sudo reboot
+# ... after reboot ...
+ls -la /var/lib/myservice/heartbeat   # still there, mtime unchanged
+
+# 4. Run the watchdog manually to confirm it passes
+sudo bash /path/to/watchdog.sh
+```
+
+## Why Not Use `tmpfiles.d` Instead
+
+systemd-tmpfiles can recreate files in `/run` at boot via a `f /run/<name> 0644 root root - -` entry. That works, but it's the wrong tool for this problem: a boot-created empty file has the boot time as its mtime, which defeats the watchdog's age check. The watchdog would see a fresh heartbeat after every reboot even if the monitored job hasn't actually run in days.
+
+Keep `/run` for true runtime state (PIDs, sockets, locks). Put success markers on persistent storage.
+
+## Related
+
+- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) — another class of post-reboot surprise
+- [rsync Backup Patterns](../02-selfhosting/storage-backup/rsync-backup-patterns.md) — reusable backup script patterns

05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md

@@ -1,3 +1,12 @@
+---
+title: "Docker & Caddy Recovery After Reboot (Fedora + SELinux)"
+domain: troubleshooting
+category: general
+tags: [docker, caddy, selinux, fedora, reboot, majorlab]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Docker & Caddy Recovery After Reboot (Fedora + SELinux)
 
 ## 🛑 Problem

05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md

@@ -0,0 +1,84 @@
+---
+title: "n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix"
+domain: troubleshooting
+category: docker
+tags: [n8n, caddy, reverse-proxy, docker, express]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix
+
+## The Problem
+
+When running n8n behind a reverse proxy (Caddy, Nginx, Traefik), the logs fill with:
+
+```
+ValidationError: The 'X-Forwarded-For' header is set but the Express 'trust proxy' setting is false (default).
+This could indicate a misconfiguration which would prevent express-rate-limit from accurately identifying users.
+```
+
+This means n8n's Express rate limiter sees every request as coming from the proxy's internal IP, not the real client. Rate limiting and audit logging both break.
+
+## Why `N8N_TRUST_PROXY=true` Isn't Enough
+
+Older n8n versions accepted `N8N_TRUST_PROXY=true` to trust proxy headers. Newer versions (1.x+) use Express's `trust proxy` setting, which requires knowing *how many* proxy hops to trust. Without `N8N_PROXY_HOPS`, Express ignores the `X-Forwarded-For` header entirely even if `N8N_TRUST_PROXY=true` is set.
+
+## The Fix
+
+Add `N8N_PROXY_HOPS=1` to your n8n environment:
+
+### Docker Compose
+
+```yaml
+services:
+  n8n:
+    image: docker.n8n.io/n8nio/n8n:latest
+    environment:
+      - N8N_HOST=n8n.example.com
+      - N8N_PROTOCOL=https
+      - N8N_TRUST_PROXY=true
+      - N8N_PROXY_HOPS=1       # <-- Add this
+```
+
+Set `N8N_PROXY_HOPS` to the number of reverse proxies between the client and n8n:
+- **1** — single proxy (Caddy/Nginx directly in front of n8n)
+- **2** — two proxies (e.g., Cloudflare → Caddy → n8n)
+
+### Recreate the Container
+
+```bash
+cd /opt/n8n  # or wherever your compose file lives
+docker compose down
+docker compose up -d
+```
+
+If you get a container name conflict:
+
+```bash
+docker rm -f n8n-n8n-1
+docker compose up -d
+```
+
+## Verifying the Fix
+
+Check the logs after restart:
+
+```bash
+docker logs --since 5m n8n-n8n-1 2>&1 | grep -i "forwarded\|proxy\|ValidationError"
+```
+
+If the fix worked, there should be zero `ValidationError` lines. A clean startup looks like:
+
+```
+n8n ready on ::, port 5678
+Version: 2.14.2
+Editor is now accessible via:
+https://n8n.example.com
+```
+
+## Key Notes
+
+- Keep `N8N_TRUST_PROXY=true` alongside `N8N_PROXY_HOPS` — both are needed.
+- The `mount of type volume should not define bind option` warning from Docker Compose when using `:z` (SELinux) volume labels is cosmetic and can be ignored.
+- If n8n reports "Last session crashed" after a `docker rm -f` recreation, this is expected — the old container was force-killed, so n8n sees it as a crash. It recovers automatically.

05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md

@@ -0,0 +1,82 @@
+---
+title: "Nextcloud AIO Container Unhealthy for 20 Hours After Nightly Update"
+domain: troubleshooting
+category: docker
+tags: [nextcloud, docker, healthcheck, netdata, php-fpm, aio]
+status: published
+created: 2026-03-28
+updated: 2026-03-28
+---
+
+# Nextcloud AIO Container Unhealthy for 20 Hours After Nightly Update
+
+## Symptom
+
+Netdata alert `docker_nextcloud_unhealthy` fired on majorlab and stayed in Warning for 20 hours. The `nextcloud-aio-nextcloud` container was running but its Docker healthcheck kept failing. No user-facing errors were visible in `nextcloud.log`.
+
+## Investigation
+
+### Timeline (2026-03-27, all UTC)
+
+| Time | Event |
+|---|---|
+| 04:00 | Nightly backup script started, mastercontainer update kicked off |
+| 04:03 | `nextcloud-aio-nextcloud` container recreated |
+| 04:05 | Backup finished |
+| 07:25 | Mastercontainer logged "Initial startup of Nextcloud All-in-One complete!" (3h20m delay) |
+| 10:22 | First entry in `nextcloud.log` (deprecation warnings only — no errors) |
+| 04:00 (Mar 28) | Next nightly backup replaced the container; new container came up healthy in ~25 minutes |
+
+### Key findings
+
+- **No image update** — the container image dated to Feb 26, so this was not caused by a version change.
+- **No app-level errors** — `nextcloud.log` contained only `files_rightclick` deprecation warnings (level 3). No level 2/4 entries.
+- **PHP-FPM never stabilized** — the healthcheck (`/healthcheck.sh`) tests `nc -z 127.0.0.1 9000` (PHP-FPM). The container was running but FPM wasn't responding to the port check.
+- **6-hour log gap** — no `nextcloud.log` entries between container start (04:03) and first log (10:22), suggesting the AIO init scripts (occ upgrade, app updates, cron jobs) ran for hours before the app became partially responsive.
+- **RestartCount: 0** — the container never restarted on its own. It sat there unhealthy for the full 20 hours.
+- **Disk space fine** — 40% used on `/`.
+
+### Healthcheck details
+
+```bash
+#!/bin/bash
+# /healthcheck.sh inside nextcloud-aio-nextcloud
+nc -z "$POSTGRES_HOST" "$POSTGRES_PORT" || exit 0  # postgres down = pass (graceful)
+nc -z 127.0.0.1 9000 || exit 1                      # PHP-FPM down = fail
+```
+
+If PostgreSQL is unreachable, the check passes (exits 0). The only failure path is PHP-FPM not listening on port 9000.
+
+## Root Cause
+
+The AIO nightly update cycle recreated the container, but the startup/migration process hung or ran extremely long, preventing PHP-FPM from fully initializing. The container sat in this state for 20 hours with no self-recovery mechanism until the next nightly cycle replaced it.
+
+The exact migration or occ command that stalled could not be confirmed — the old container's entrypoint logs were lost when the Mar 28 backup cycle replaced it.
+
+## Fix
+
+Two changes deployed on 2026-03-28:
+
+### 1. Dedicated Netdata alarm with lenient window
+
+Split `nextcloud-aio-nextcloud` into its own Netdata alarm (`docker_nextcloud_unhealthy`) with a 10-minute lookup and 10-minute delay, separate from the general container alarm. See [Tuning Netdata Docker Health Alarms](../../02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md).
+
+### 2. Watchdog cron for auto-restart
+
+Deployed `/etc/cron.d/nextcloud-health-watchdog` on majorlab:
+
+```bash
+*/15 * * * * root docker inspect --format={{.State.Health.Status}} nextcloud-aio-nextcloud 2>/dev/null | grep -q unhealthy && [ "$(docker inspect --format={{.State.StartedAt}} nextcloud-aio-nextcloud | xargs -I{} date -d {} +\%s)" -lt "$(date -d "1 hour ago" +\%s)" ] && docker restart nextcloud-aio-nextcloud && logger -t nextcloud-watchdog "Restarted unhealthy nextcloud-aio-nextcloud"
+```
+
+- Checks every 15 minutes
+- Only restarts if the container has been running >1 hour (avoids interfering with normal startup)
+- Logs to syslog: `journalctl -t nextcloud-watchdog`
+
+This caps future unhealthy outages at ~1 hour instead of persisting until the next nightly cycle.
+
+## See Also
+
+- [Tuning Netdata Docker Health Alarms](../../02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md)
+- [Debugging Broken Docker Containers](../../02-selfhosting/docker/debugging-broken-docker-containers.md)
+- [Docker Healthchecks](../../02-selfhosting/docker/docker-healthchecks.md)

05-troubleshooting/fedora-networking-kernel-recovery.md

@@ -0,0 +1,141 @@
+---
+title: "Fedora Networking & Kernel Troubleshooting"
+domain: troubleshooting
+category: networking
+tags: [fedora, networking, kernel, grub, nmcli, troubleshooting]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+
+# Fedora Networking & Kernel Troubleshooting
+
+Two common issues on the MajorsHouse Fedora fleet (majorlab, majorhome): network connectivity dropping after updates or reboots, and kernel upgrades that break things. These are the quick fixes and the deeper recovery paths.
+
+## Networking Drops After Reboot or Update
+
+### Quick Fix
+
+If a Fedora box loses network connectivity after a reboot or `dnf upgrade`, NetworkManager may not have brought the connection back up automatically:
+
+```bash
+nmcli connection up "Wired connection 1"
+```
+
+This re-activates the default wired connection. If the connection name differs on your system:
+
+```bash
+# List all known connections
+nmcli connection show
+
+# Bring up by name
+nmcli connection up "your-connection-name"
+```
+
+### Why This Happens
+
+- NetworkManager may not auto-activate a connection if it was configured as manual or if the profile was reset during an upgrade.
+- Kernel updates can temporarily break network drivers, especially on hardware with out-of-tree modules. The new kernel loads, the old driver doesn't match, and the NIC doesn't come up.
+- On headless servers (like majorlab and majorhome), there's no desktop network applet to reconnect — it stays down until you fix it via console or IPMI.
+
+### Make It Persistent
+
+Ensure the connection auto-activates on boot:
+
+```bash
+# Check current autoconnect setting
+nmcli connection show "Wired connection 1" | grep autoconnect
+
+# Enable if not set
+nmcli connection modify "Wired connection 1" connection.autoconnect yes
+```
+
+## Kernel Issues — Booting an Older Kernel
+
+When a new kernel causes problems (network, storage, GPU, or boot failures), boot into the previous working kernel via GRUB.
+
+### At the GRUB Menu
+
+1. Reboot the machine.
+2. Hold **Shift** (BIOS) or press **Esc** (UEFI) to show the GRUB menu.
+3. Select **Advanced options** or an older kernel entry.
+4. Boot into the working kernel.
+
+### From the Command Line (Headless)
+
+If you have console access but no GRUB menu:
+
+```bash
+# List installed kernels
+sudo grubby --info=ALL | grep -E "^(index|kernel|title)"
+
+# Set the previous kernel as default (by index)
+sudo grubby --set-default-index=1
+
+# Or set by kernel path
+sudo grubby --set-default=/boot/vmlinuz-6.19.9-200.fc43.x86_64
+
+# Reboot into it
+sudo reboot
+```
+
+### Remove a Bad Kernel
+
+Once you've confirmed the older kernel works:
+
+```bash
+# Remove the broken kernel
+sudo dnf remove kernel-core-6.19.10-200.fc43.x86_64
+
+# Verify GRUB updated
+sudo grubby --default-kernel
+```
+
+### Prevent Auto-Updates From Reinstalling It
+
+If the same kernel version keeps coming back and keeps breaking:
+
+```bash
+# Temporarily exclude it from updates
+sudo dnf upgrade --exclude=kernel*
+
+# Or pin in dnf.conf
+echo "excludepkgs=kernel*" | sudo tee -a /etc/dnf/dnf.conf
+```
+
+Remove the exclusion once a fixed kernel version is released.
+
+## Quick Diagnostic Commands
+
+```bash
+# Check current kernel
+uname -r
+
+# Check network status
+nmcli general status
+nmcli device status
+ip addr show
+
+# Check if NetworkManager is running
+systemctl status NetworkManager
+
+# Check recent kernel/network errors
+journalctl -b -p err | grep -iE "kernel|network|eth|ens|nm"
+
+# Check which kernels are installed
+rpm -qa kernel-core | sort -V
+```
+
+## Gotchas & Notes
+
+- **Always have console access** (IPMI, physical KVM, or Proxmox console) for headless servers before doing kernel updates. If the new kernel breaks networking, SSH won't save you.
+- **Fedora keeps 3 kernels by default** (`installonly_limit=3` in `/etc/dnf/dnf.conf`). If you need more fallback options, increase this number before upgrading.
+- **Test kernel updates on one server first.** Update majorlab, confirm it survives a reboot, then update majorhome.
+- **`grubby` is Fedora's preferred tool** for managing GRUB entries. Avoid editing `grub.cfg` directly.
+
+Reference: [Fedora — Working with the GRUB 2 Boot Loader](https://docs.fedoraproject.org/en-US/fedora/latest/system-administrators-guide/kernel-module-driver-configuration/Working_with_the_GRUB_2_Boot_Loader/)
+
+## See Also
+
+- [docker-caddy-selinux-post-reboot-recovery](docker-caddy-selinux-post-reboot-recovery.md)
+- [managing-linux-services-systemd-ansible](../01-linux/process-management/managing-linux-services-systemd-ansible.md)

05-troubleshooting/gemini-cli-manual-update.md

@@ -1,3 +1,12 @@
+---
+title: "Gemini CLI: Manual Update Guide"
+domain: troubleshooting
+category: general
+tags: [gemini, cli, npm, update, google]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # 🛠️ Gemini CLI: Manual Update Guide
 
 If the automatic update fails or you need to force a specific version of the Gemini CLI, use these steps.

05-troubleshooting/gitea-runner-boot-race-network-target.md

@@ -1,3 +1,12 @@
+---
+title: "Gitea Actions Runner: Boot Race Condition Fix"
+domain: troubleshooting
+category: general
+tags: [gitea, systemd, boot, dns, ci-cd]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Gitea Actions Runner: Boot Race Condition Fix
 
 If your `gitea-runner` (act_runner) service fails to start on boot — crash-looping and eventually hitting systemd's restart rate limit — the service is likely starting before DNS is available.

05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md

@@ -1,3 +1,12 @@
+---
+title: "Qwen2.5-14B OOM on RTX 3080 Ti (12GB)"
+domain: troubleshooting
+category: gpu-display
+tags: [gpu, vram, oom, qwen, cuda, fine-tuning]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Qwen2.5-14B OOM on RTX 3080 Ti (12GB)
 
 ## Problem
@@ -52,7 +61,3 @@ If you have no choice but to try 14B training:
 Keep your NVIDIA drivers and CUDA toolkit updated. On Windows (MajorRig), ensure WSL2 has sufficient memory allocation in `.wslconfig`.
 
 ---
-
-## Tags
-
-#gpu #cuda #oom #qwen #majortwin #llm #fine-tuning

05-troubleshooting/index.md

@@ -1,3 +1,7 @@
+---
+created: 2026-03-15T06:37
+updated: 2026-04-17T09:57
+---
 # 🔧 General Troubleshooting
 
 Practical fixes for common Linux, networking, and application problems.
@@ -10,13 +14,22 @@ Practical fixes for common Linux, networking, and application problems.
 - [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](networking/fail2ban-imap-self-ban-mail-client.md)
 - [firewalld: Mail Ports Wiped After Reload](networking/firewalld-mail-ports-reset.md)
 - [Tailscale SSH: Unexpected Re-Authentication Prompt](networking/tailscale-ssh-reauth-prompt.md)
+- [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md)
 - [ISP SNI Filtering & Caddy](isp-sni-filtering-caddy.md)
 - [yt-dlp YouTube JS Challenge Fix](yt-dlp-fedora-js-challenge.md)
+- [wget/curl: URLs with Special Characters Fail in Bash](wget-url-special-characters.md)
+
+## ⚙️ Ansible & Fleet Management
+- [SSH Timeout During dnf upgrade on Fedora Hosts](ansible-ssh-timeout-dnf-upgrade.md)
+- [Vault Password File Missing](ansible-vault-password-file-missing.md)
+- [ansible.cfg Ignored on WSL2 Windows Mounts](ansible-wsl2-world-writable-mount-ignores-cfg.md)
 
 ## 📦 Docker & Systems
 - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md)
 - [Gitea Actions Runner: Boot Race Condition Fix](gitea-runner-boot-race-network-target.md)
+- [Systemd Session Scope Fails at Login (`session-cN.scope`)](systemd/session-scope-failure-at-login.md)
 - [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md)
+- [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](cron-heartbeat-tmpfs-reboot-false-alarm.md)
 
 ## 🔒 SELinux
 - [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](selinux-dovecot-vmail-context.md)
@@ -31,3 +44,4 @@ Practical fixes for common Linux, networking, and application problems.
 ## 🤖 AI / Local LLM
 - [Ollama Drops Off Tailscale When Mac Sleeps](ollama-macos-sleep-tailscale-disconnect.md)
 - [Windows OpenSSH Server (sshd) Stops After Reboot](networking/windows-sshd-stops-after-reboot.md)
+- [claude-mem Silently Fails with Claude Code 2.1+ (Empty `--setting-sources`)](claude-mem-setting-sources-empty-arg.md)

05-troubleshooting/isp-sni-filtering-caddy.md

@@ -1,3 +1,12 @@
+---
+title: "ISP SNI Filtering & Caddy Troubleshooting"
+domain: troubleshooting
+category: general
+tags: [isp, sni, caddy, tls, dns, cloudflare]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # ISP SNI Filtering & Caddy Troubleshooting
 
 ## 🛑 Problem

05-troubleshooting/macos-mirrored-notification-alert-loop.md

@@ -0,0 +1,95 @@
+---
+title: "macOS Repeating Alert Tone from Mirrored iPhone Notification"
+domain: troubleshooting
+category: general
+tags: [macos, iphone, notifications, continuity, audio]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# macOS Repeating Alert Tone from Mirrored iPhone Notification
+
+## Overview
+On macOS, an unacknowledged iPhone notification can be mirrored to a Mac via iPhone Mirroring or Continuity and loop indefinitely — even after the alert is dismissed on the iPhone. The sound plays through the Mac's built-in speakers with no visible notification banner.
+
+## Symptoms
+- Repeating alarm tone coming from Mac speakers at regular intervals
+- No visible notification in Notification Center or as a banner
+- Sound starts when the Mac is opened or woken
+- iPhone is silent
+- iPhone not connected via Bluetooth audio
+
+## Root Cause
+macOS's `ToneLibrary` framework (`TLAlertQueuePlayerController`) loops iPhone alert ringtones via `NotificationCenter` when a mirrored notification has not been dismissed on the Mac side. The iPhone side may show the alert as acknowledged, but the Mac maintains its own notification state independently.
+
+The sound file being looped will be an iPhone ringtone from:
+```
+/System/Library/PrivateFrameworks/ToneLibrary.framework/Resources/Ringtones/
+```
+
+## Diagnosis
+
+To confirm this is the cause, run:
+```bash
+log stream --predicate 'process == "NotificationCenter"' --style compact
+```
+Wait for the sound to fire. Look for a line like:
+```
+URL = file:///System/Library/PrivateFrameworks/ToneLibrary.framework/Resources/Ringtones/<name>.m4r
+```
+If `TLAlertQueuePlayerController` appears in the output alongside a `.m4r` ringtone URL, this is your issue.
+
+### Full Diagnostic Sequence
+If the above isn't conclusive, work through these in order:
+
+**1. Broad scan:**
+```bash
+log stream --predicate 'eventMessage contains "sound" OR eventMessage contains "alert" OR eventMessage contains "notification"' --style compact
+```
+
+**2. CoreAudio scan (catches audio playback):**
+```bash
+log stream --predicate 'subsystem == "com.apple.coreaudio" OR eventMessage contains "AudioSession" OR eventMessage contains "AVAudio"' --style compact
+```
+Look for `NotificationCenter` with `contentType = 'soun'` and `Route = built-in speakers`.
+
+**3. Confirm with NotificationCenter filter:**
+```bash
+log stream --predicate 'process == "NotificationCenter"' --style compact
+```
+
+## Fix
+
+### Immediate
+Kill and restart NotificationCenter:
+```bash
+killall -9 NotificationCenter
+```
+macOS will relaunch it automatically. The looping sound will stop immediately.
+
+### Proper Dismissal
+Open Notification Center on the Mac (click the clock in the menu bar) and dismiss any queued notifications from the offending app. If the source notification is still pending on the iPhone, dismiss it there as well.
+
+## Prevention
+
+### Option A — Disable sound for the app on Mac
+Go to **System Settings → Notifications → [App Name]** and either:
+- Turn off **Play sound for notifications**, or
+- Turn off notifications entirely for the app on the Mac
+
+The iPhone will still alert normally.
+
+### Option B — Dismiss on both devices
+When a high-priority alert fires on iPhone, check both the iPhone and the Mac's Notification Center to ensure both sides are cleared.
+
+## Notes
+- This behaviour can be triggered by any iPhone app whose notifications are mirrored to the Mac via Continuity or iPhone Mirroring, not just apps with alarm-style alerts.
+- The Mac maintains its own notification state independently of the iPhone — dismissing on one device does not guarantee dismissal on the other.
+- Apps with persistent or repeating alert styles (such as health monitors, timers, or messaging apps) are most likely to trigger this issue.
+- Unrelated: Ivory (Mastodon client) may show excessive badge update calls in logs on startup — this is a known Ivory bug and not related to audio playback.
+
+## Tags
+`macos` `notifications` `iphone-mirroring` `continuity` `tonelibrary` `notificationcenter` `diagnostic` `audio`
+
+## See Also
+- 2026-03-30-MajorAir-CGM-Alert-Loop (diagnostic journal entry)

05-troubleshooting/majwiki-setup-and-pipeline.md

@@ -1,8 +1,16 @@
 ---
-title: "MajorWiki Setup & Publishing Pipeline"
+title: MajorWiki Setup & Publishing Pipeline
 domain: troubleshooting
-tags: [mkdocs, obsidian, gitea, docker, self-hosting]
+category: general
-date: 2026-03-11
+tags:
+  - mkdocs
+  - obsidian
+  - gitea
+  - docker
+  - self-hosting
+status: published
+created: 2026-03-11
+updated: 2026-04-07T10:48
 ---
 
 # MajorWiki Setup & Publishing Pipeline
@@ -130,7 +138,7 @@ The Obsidian Git plugin was evaluated but dropped — too convoluted for a simpl
 
 ```bash
 cd ~/Documents/MajorVault
-git add 20-Projects/MajorTwin/08-Wiki/
+git add 30-Areas/MajorWiki/
 git commit -m "wiki: describe your changes"
 git push
 ```

05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md

@@ -1,3 +1,12 @@
+---
+title: "Mail Client Stops Receiving: Fail2ban IMAP Self-Ban"
+domain: troubleshooting
+category: networking
+tags: [fail2ban, imap, dovecot, email, self-ban]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Mail Client Stops Receiving: Fail2ban IMAP Self-Ban
 
 ## 🛑 Problem

05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md

@@ -1,3 +1,12 @@
+---
+title: "Apache Outage: Fail2ban Self-Ban + Missing iptables Rules"
+domain: troubleshooting
+category: networking
+tags: [fail2ban, apache, iptables, self-ban, outage]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Apache Outage: Fail2ban Self-Ban + Missing iptables Rules
 
 ## 🛑 Problem

05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md

@@ -0,0 +1,167 @@
+---
+title: "Fail2ban & UFW Rule Bloat: 30k Rules Slowing Down a VPS"
+domain: troubleshooting
+category: networking
+tags: [fail2ban, ufw, nftables, vps, performance]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# Fail2ban & UFW Rule Bloat: 30k Rules Slowing Down a VPS
+
+## 🛑 Problem
+
+A small VPS (1–2 GB RAM) running Fail2ban with permanent bans (`bantime = -1`) gradually accumulates thousands of UFW DENY rules or nftables entries. Over time this causes:
+
+- High memory usage from Fail2ban (100+ MB RSS)
+- Bloated nftables ruleset (30k+ rules) — every incoming packet must traverse the full list
+- Netdata alerts flapping on RAM/swap thresholds
+- Degraded packet processing performance
+
+---
+
+## 🔍 Diagnosis
+
+### Step 1 — Check Fail2ban memory and thread count
+
+```bash
+grep -E "VmRSS|VmSwap|Threads" /proc/$(pgrep -ox fail2ban-server)/status
+```
+
+On a small VPS, Fail2ban RSS over 80 MB is a red flag. Thread count scales with jail count (roughly 2 threads per jail + overhead).
+
+---
+
+### Step 2 — Count nftables/UFW rules
+
+```bash
+# Total drop/reject rules in nftables
+nft list ruleset | grep -c "reject\|drop"
+
+# UFW rule file size
+wc -l /etc/ufw/user.rules
+```
+
+A healthy UFW setup has 10–30 rules. Thousands means manual `ufw deny` commands or permanent Fail2ban bans have accumulated.
+
+---
+
+### Step 3 — Identify dead jails
+
+```bash
+for jail in $(fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
+  total=$(fail2ban-client status $jail | grep "Total banned" | awk '{print $NF}')
+  echo "$jail: $total total bans"
+done
+```
+
+Jails with zero total bans are dead weight — burning threads and regex cycles for nothing.
+
+---
+
+### Step 4 — Check ban policy
+
+```bash
+grep bantime /etc/fail2ban/jail.local
+```
+
+`bantime = -1` means permanent. On a public-facing server, scanner IPs rotate constantly — permanent bans just pile up with no benefit.
+
+---
+
+## ✅ Solution
+
+### Fix 1 — Disable dead jails
+
+Edit `/etc/fail2ban/jail.local` and set `enabled = false` for any jail with zero historical bans.
+
+### Fix 2 — Switch to time-limited bans
+
+```ini
+[DEFAULT]
+bantime = 30d
+
+[recidive]
+bantime = 90d
+```
+
+30 days is long enough to block active campaigns; repeat offenders get 90 days via recidive. Scanner IPs rarely persist beyond a week.
+
+### Fix 3 — Flush accumulated bans
+
+```bash
+fail2ban-client unban --all
+```
+
+### Fix 4 — Reset bloated UFW rules
+
+**Back up first:**
+
+```bash
+cp /etc/ufw/user.rules /etc/ufw/user.rules.bak
+cp /etc/ufw/user6.rules /etc/ufw/user6.rules.bak
+```
+
+**Reset and re-add only legitimate ALLOW rules:**
+
+```bash
+ufw --force reset
+ufw default deny incoming
+ufw default allow outgoing
+ufw allow 443/tcp
+ufw allow 80/tcp
+ufw allow in on tailscale0 to any port 22 comment "SSH via Tailscale"
+# Add any other ALLOW rules specific to your server
+ufw --force enable
+```
+
+**Restart Fail2ban** so it re-creates its nftables chains:
+
+```bash
+systemctl restart fail2ban
+```
+
+---
+
+## 🔁 Why This Happens
+
+| Cause | Effect |
+|---|---|
+| `bantime = -1` (permanent) | Banned IP list grows forever; nftables rules never expire |
+| Manual `ufw deny from <IP>` | Each adds a persistent rule to `user.rules`; survives reboots |
+| Many jails with no hits | Each jail spawns 2+ threads, runs regex against logs continuously |
+| Small VPS (1–2 GB RAM) | Fail2ban + nftables overhead becomes significant fraction of total RAM |
+
+---
+
+## ⚠️ Key Notes
+
+- **Deleting UFW rules one-by-one is impractical** at scale — `ufw delete` with 30k rules takes hours. A full reset + re-add is the only efficient path.
+- **`ufw --force reset` also resets `before.rules` and `after.rules`** — UFW auto-backs these up, but verify your custom chains if any exist.
+- **After flushing bans, expect a brief spike in 4xx responses** as scanners that were previously blocked hit Apache again. Fail2ban will re-ban them within minutes.
+- **The Netdata `web_log_1m_successful` alert may fire** during this window — it will self-clear once bans repopulate.
+
+---
+
+## 🔎 Quick Diagnostic Commands
+
+```bash
+# Fail2ban memory usage
+grep -E "VmRSS|VmSwap|Threads" /proc/$(pgrep -ox fail2ban-server)/status
+
+# Count nftables rules
+nft list ruleset | grep -c "reject\|drop"
+
+# UFW rule count
+ufw status numbered | tail -1
+
+# List all jails with ban counts
+for jail in $(fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
+  banned=$(fail2ban-client status $jail | grep "Currently banned" | awk '{print $NF}')
+  total=$(fail2ban-client status $jail | grep "Total banned" | awk '{print $NF}')
+  echo "$jail: $banned current / $total total"
+done
+
+# Flush all bans
+fail2ban-client unban --all
+```

05-troubleshooting/networking/firewalld-mail-ports-reset.md

@@ -1,3 +1,12 @@
+---
+title: "firewalld: Mail Ports Wiped After Reload (IMAP + Webmail Outage)"
+domain: troubleshooting
+category: networking
+tags: [firewalld, mail, imap, fedora, ports]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # firewalld: Mail Ports Wiped After Reload (IMAP + Webmail Outage)
 
 If IMAP, SMTP, and webmail all stop working simultaneously on a Fedora/RHEL mail server, firewalld may have reloaded and lost its mail port configuration.

05-troubleshooting/networking/tailscale-ssh-reauth-prompt.md

@@ -1,3 +1,12 @@
+---
+title: "Tailscale SSH: Unexpected Re-Authentication Prompt"
+domain: troubleshooting
+category: networking
+tags: [tailscale, ssh, authentication, vpn]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # Tailscale SSH: Unexpected Re-Authentication Prompt
 
 If a Tailscale SSH connection unexpectedly presents a browser authentication URL mid-session, the first instinct is to check the ACL policy. However, this is often a one-off Tailscale hiccup rather than a misconfiguration.
@@ -62,5 +71,5 @@ No auth prompt + `ok` output = resolved. Note that this test is only meaningful
 
 ## Related
 
-- [[Network Overview]] — Tailscale fleet inventory and SSH access model
+- Network Overview — Tailscale fleet inventory and SSH access model
-- [[SSH-Aliases]] — Fleet SSH access shortcuts
+- SSH-Aliases — Fleet SSH access shortcuts

05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md

@@ -0,0 +1,93 @@
+---
+title: "Windows OpenSSH: WSL as Default Shell Breaks Remote Commands"
+domain: troubleshooting
+category: networking
+tags:
+  - windows
+  - openssh
+  - wsl
+  - ssh
+  - majorrig
+  - powershell
+status: published
+created: 2026-04-03
+updated: 2026-04-07T21:55
+---
+
+# Windows OpenSSH: WSL as Default Shell Breaks Remote Commands
+
+## Problem
+
+SSH remote commands fail with:
+
+```
+Invalid command line argument: -c
+Please use 'wsl.exe --help' to get a list of supported arguments.
+```
+
+This happens on **every** remote command — `ssh-copy-id`, `ssh user@host "command"`, `scp`, etc. Interactive SSH (no command) may still work if it drops into WSL.
+
+## Cause
+
+Windows OpenSSH's default shell is set to `C:\Windows\System32\wsl.exe`. When SSH executes a remote command, it invokes:
+
+```
+<default_shell> -c "<command>"
+```
+
+But `wsl.exe` does not accept the `-c` flag. It expects `-e` for command execution, or no flags for an interactive session. Since OpenSSH hardcodes `-c`, every remote command fails.
+
+## Fix — Option 1: Use `bash.exe` (Recommended)
+
+`bash.exe` is a WSL shim that **does** accept the `-c` flag. This gives you a Linux-first SSH experience where both interactive sessions and remote commands work natively.
+
+```powershell
+# Find the actual path to bash.exe (it varies by install)
+Get-Command bash.exe | Select-Object Source
+
+# Set it as the default shell (elevated PowerShell)
+Set-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Users\<username>\AppData\Local\Microsoft\WindowsApps\bash.exe"
+Restart-Service sshd
+```
+
+> **Note:** `bash.exe` may not be at `C:\Windows\System32\bash.exe` on all installs. Always verify the path with `Get-Command` first — the Windows Store WSL install places it under `AppData\Local\Microsoft\WindowsApps\`.
+
+### After the fix (bash.exe)
+
+- Interactive SSH sessions land directly in your WSL distro
+- Remote SSH commands execute in WSL's bash — Linux commands work natively
+- `ssh user@host "uname -s"` returns `Linux`
+
+## Fix — Option 2: Revert to PowerShell
+
+If you need Windows-native command execution over SSH (e.g., for Windows-targeted Ansible or remote PowerShell administration), set the default shell back to PowerShell:
+
+```powershell
+Set-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
+Restart-Service sshd
+```
+
+### After the fix (PowerShell)
+
+- Remote SSH commands execute via PowerShell
+- To run Linux commands, prefix with `wsl`:
+  ```bash
+  ssh user@host "wsl bash -c 'cd /mnt/d/project && git pull'"
+  ```
+- Interactive SSH sessions land in PowerShell (type `wsl` to enter Linux)
+- `ssh-copy-id` still won't work for WSL's `authorized_keys` — Windows OpenSSH reads from `C:\Users\<user>\.ssh\authorized_keys`, not the WSL home directory
+
+## Key Notes
+
+- This registry key (`HKLM:\SOFTWARE\OpenSSH\DefaultShell`) is the **only** supported way to change the OpenSSH default shell on Windows
+- The change persists across reboots and Windows Updates
+- `wsl.exe` does **not** support `-c` — never use it as the default shell
+- `bash.exe` **does** support `-c` — use it for a Linux-first SSH experience
+- The path to `bash.exe` varies by install method — always verify with `Get-Command bash.exe`
+- Tools like Ansible, `scp`, `rsync`, and `ssh-copy-id` all depend on `-c` working
+- If the shell path in the registry doesn't exist on disk, sshd will reject the user entirely with `User <name> not allowed because shell <path> does not exist` — check `Get-WinEvent -LogName OpenSSH/Operational` to diagnose
+
+## Related
+
+- [Windows OpenSSH Server (sshd) Stops After Reboot](windows-sshd-stops-after-reboot.md) — sshd service startup issues
+- [Microsoft Docs: OpenSSH DefaultShell](https://learn.microsoft.com/en-us/windows-server/administration/openssh/openssh-server-configuration#configuring-the-default-shell-for-openssh-in-windows)

05-troubleshooting/networking/windows-sshd-stops-after-reboot.md

@@ -1,3 +1,17 @@
+---
+title: Windows OpenSSH Server (sshd) Stops After Reboot
+domain: troubleshooting
+category: networking
+tags:
+  - windows
+  - openssh
+  - sshd
+  - reboot
+  - majorrig
+status: published
+created: 2026-04-02
+updated: 2026-04-07T21:58
+---
 # Windows OpenSSH Server (sshd) Stops After Reboot
 
 ## 🛑 Problem
@@ -49,7 +63,7 @@ The Windows OpenSSH Server is installed as a Windows Feature (`Add-WindowsCapabi
 - **This is a Windows-side issue** — WSL2 itself is unaffected. The service must be started and configured from Windows, not from within WSL2.
 - **Elevated PowerShell required** — `Start-Service` and `Set-Service` for sshd will return "Access is denied" if run without Administrator privileges.
 - **Port 2222 was retired (2026-03-25)** — the bypass port 2222 on MajorRig is no longer in use. The entire fleet now uses port 22 uniformly after the Tailscale SSH auth fix. Only port 22 needs to be verified when troubleshooting sshd.
-- **Default shell still works once fixed** — MajorRig's sshd is configured to use `C:\Windows\System32\wsl.exe` as the default shell, dropping SSH sessions directly into WSL2/Bash. This config is preserved across service restarts.
+- **Default shell still works once fixed** — MajorRig's sshd is configured to use `bash.exe` (WSL shim) as the default shell, dropping SSH sessions directly into WSL2/Bash. This config is preserved across service restarts. See [WSL default shell troubleshooting](windows-openssh-wsl-default-shell-breaks-remote-commands.md) for why `bash.exe` is used instead of `wsl.exe`.
 
 ---
 

05-troubleshooting/obsidian-cache-hang-recovery.md

@@ -1,11 +1,11 @@
 ---
-tags:
+title: "Obsidian Vault Recovery — Loading Cache Hang"
-  - obsidian
+domain: troubleshooting
-  - troubleshooting
+category: general
-  - windows
+tags: [obsidian, troubleshooting, windows, majortwin]
-  - majortwin
+status: published
-created: '2026-03-11'
+created: 2026-03-11
-status: resolved
+updated: 2026-04-02
 ---
 # Obsidian Vault Recovery — Loading Cache Hang
 

05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md

@@ -65,4 +65,4 @@ The display can still sleep — only system sleep needs to be off for Ollama and
 
 ## See Also
 
-- [[MajorMac]] — device config and known issues
+- MajorMac — device config and known issues

05-troubleshooting/security/apache-dirscan-fail2ban-jail.md

@@ -0,0 +1,122 @@
+---
+title: "Custom Fail2ban Jail: Apache Directory Scanning & Junk Methods"
+domain: troubleshooting
+category: security
+tags: [fail2ban, apache, security, bots, wordpress]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# Custom Fail2ban Jail: Apache Directory Scanning & Junk Methods
+
+## 🛑 Problem
+
+Bots and vulnerability scanners enumerate WordPress directories (`/wp-admin/`, `/wp-includes/`, `/wp-content/`), probe for access-denied paths, or send junk HTTP methods (e.g., `YQEILVHZ`, `DUTEDCEM`). These generate Apache error log entries but are not caught by any default Fail2ban jail:
+
+- `AH01276` — directory index forbidden (autoindex:error)
+- `AH01630` — client denied by server configuration (authz_core:error)
+- `AH00135` — invalid method in request (core:error)
+
+The result is a low success ratio on Netdata's `web_log_1m_successful` metric and wasted server resources processing scanner requests.
+
+---
+
+## ✅ Solution
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/apache-dirscan.conf`:
+
+```ini
+# Fail2ban filter for Apache scanning/probing
+# Catches: directory enumeration (AH01276), access denied (AH01630), invalid methods (AH00135)
+
+[Definition]
+failregex = ^\[.*\] \[autoindex:error\] \[pid \d+\] \[client <HOST>:\d+\] AH01276:
+            ^\[.*\] \[authz_core:error\] \[pid \d+\] \[client <HOST>:\d+\] AH01630:
+            ^\[.*\] \[core:error\] \[pid \d+\] \[client <HOST>:\d+\] AH00135:
+
+ignoreregex =
+```
+
+### Step 2 — Add the jail
+
+Add to `/etc/fail2ban/jail.local`:
+
+```ini
+[apache-dirscan]
+enabled = true
+port = http,https
+filter = apache-dirscan
+logpath = /var/log/apache2/error.log
+maxretry = 3
+findtime = 60
+```
+
+Three hits in 60 seconds is aggressive enough to catch active scanners while avoiding false positives from legitimate 403s.
+
+### Step 3 — Test the regex
+
+```bash
+fail2ban-regex /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-dirscan.conf
+```
+
+This shows match counts per regex line and any missed lines.
+
+### Step 4 — Reload Fail2ban
+
+```bash
+fail2ban-client reload
+fail2ban-client status apache-dirscan
+```
+
+---
+
+## 🔍 What Each Pattern Catches
+
+| Error Code | Apache Module | Trigger |
+|---|---|---|
+| `AH01276` | `autoindex:error` | Bot requests a directory with no index file and `Options -Indexes` is set. Classic WordPress/CMS directory enumeration. |
+| `AH01630` | `authz_core:error` | Request denied by `<Directory>` or `<Location>` rules (e.g., probing `/wp-content/plugins/`). |
+| `AH00135` | `core:error` | Request uses a garbage HTTP method that Apache can't parse. Scanners use these to fingerprint servers. |
+
+---
+
+## 🔁 Why Default Jails Miss This
+
+| Default Jail | What It Catches | Gap |
+|---|---|---|
+| `apache-badbots` | Bad User-Agent strings in access log | Doesn't look at error log; many scanners use normal UAs |
+| `apache-botsearch` | 404s for common exploit paths | Only matches access log 404s, not error log entries |
+| `apache-noscript` | Requests for non-existent scripts | Narrow regex, doesn't cover directory probes |
+| `apache-overflows` | Long request URIs | Only catches buffer overflow attempts |
+| `apache-invaliduri` | `AH10244` invalid URI encoding | Different error code — catches URL-encoded traversal, not directory scanning |
+
+The `apache-dirscan` filter fills the gap by monitoring the error log for the three most common scanner signatures that slip through all default jails.
+
+---
+
+## ⚠️ Key Notes
+
+- **`logpath` must point to the error log**, not the access log. All three patterns are logged to `error.log`.
+- **Adjust `logpath`** for your distribution: Debian/Ubuntu uses `/var/log/apache2/error.log`, RHEL/Fedora uses `/var/log/httpd/error_log`.
+- **The `allowipv6` warning** on reload is cosmetic (Fail2ban 1.0+) and can be ignored.
+- **Pair with `recidive`** to escalate repeat offenders to longer bans.
+
+---
+
+## 🔎 Quick Diagnostic Commands
+
+```bash
+# Test filter against current error log
+fail2ban-regex /var/log/apache2/error.log /etc/fail2ban/filter.d/apache-dirscan.conf
+
+# Check jail status
+fail2ban-client status apache-dirscan
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep apache-dirscan
+
+# Count current error types
+grep -c "AH01276\|AH01630\|AH00135" /var/log/apache2/error.log
+```

05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md

@@ -1,3 +1,12 @@
+---
+title: "ClamAV Safe Scheduling on Live Servers"
+domain: troubleshooting
+category: security
+tags: [clamav, cpu, nice, ionice, cron, vps]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # ClamAV Safe Scheduling on Live Servers
 
 Running `clamscan` unthrottled on a live server will peg CPU until completion. On a small VPS (1 vCPU), a full recursive scan can sustain 70–100% CPU for an hour or more, degrading or taking down hosted services.
@@ -70,4 +79,4 @@ kill <PID>
 ## Related
 
 - [ClamAV Documentation](https://docs.clamav.net/)
-- [[02-selfhosting/security/linux-server-hardening-checklist|Linux Server Hardening Checklist]]
+- [Linux Server Hardening Checklist](../../02-selfhosting/security/linux-server-hardening-checklist.md)

05-troubleshooting/selinux-dovecot-vmail-context.md

@@ -1,3 +1,12 @@
+---
+title: "SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)"
+domain: troubleshooting
+category: general
+tags: [selinux, dovecot, mail, fedora, vmail]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)
 
 If Dovecot is generating SELinux AVC denials and mail delivery or retrieval is broken on a Fedora/RHEL system with SELinux enforcing, the `/var/vmail` directory tree likely has incorrect file contexts.

05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md

@@ -1,3 +1,12 @@
+---
+title: "mdadm RAID Recovery After USB Hub Disconnect"
+domain: troubleshooting
+category: storage
+tags: [mdadm, raid, usb, storage, recovery]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # mdadm RAID Recovery After USB Hub Disconnect
 
 A software RAID array managed by mdadm can appear to catastrophically fail when the drives are connected via USB rather than SATA. The array is fine — the hub dropped out. Here's how to diagnose and recover.

05-troubleshooting/systemd/session-scope-failure-at-login.md

@@ -0,0 +1,102 @@
+---
+title: "Systemd Session Scope Fails at Login (session-cN.scope)"
+domain: troubleshooting
+category: systemd
+tags: [systemd, ssh, login, session, linux]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
+# Systemd Session Scope Fails at Login (`session-cN.scope`)
+
+After SSH login, systemd reports a failed transient unit like `session-c1.scope`. The MOTD or login banner shows `Failed Units: 1 — session-c1.scope`. This is a harmless race condition, not a real service failure.
+
+## Symptoms
+
+- Login banner or MOTD displays:
+  ```
+  Failed Units: 1
+    session-c1.scope
+  ```
+- `systemctl list-units --failed` shows one or more `session-cN.scope` units in a failed state
+- The system is otherwise healthy — no services are actually broken
+
+## What Causes It
+
+A transient session scope is created by systemd-logind every time a user logs in (SSH, console, etc.). The scope tracks the login session's process group via cgroups.
+
+The failure occurs when a login process (PID) exits before systemd can move it into the target cgroup. This is a race condition triggered by:
+
+- **Short-lived SSH connections** — automated probes, health checks, or monitoring tools that connect and immediately disconnect
+- **Sessions that disconnect before PAM completes** — network interruptions or aggressive client timeouts
+- **Cron jobs or scripts** that create transient SSH sessions
+
+systemd logs the sequence:
+
+1. `PID N vanished before we could move it to target cgroup`
+2. `No PIDs left to attach to the scope's control group, refusing.`
+3. Unit enters `failed (Result: resources)` state
+
+Because session scopes are transient (not backed by a unit file), the failed state lingers until manually cleared.
+
+## How to Diagnose
+
+### 1. Check the failed unit
+
+```bash
+systemctl status session-c1.scope
+```
+
+Look for:
+
+```
+Active: failed (Result: resources)
+```
+
+And in the log output:
+
+```
+PID <N> vanished before we could move it to target cgroup
+No PIDs left to attach to the scope's control group, refusing.
+```
+
+### 2. Confirm no real failures
+
+```bash
+systemctl list-units --failed
+```
+
+If the only failed units are `session-cN.scope` entries, the system is healthy.
+
+## Fix
+
+Reset the failed unit:
+
+```bash
+systemctl reset-failed session-c1.scope
+```
+
+To clear all failed session scopes at once:
+
+```bash
+systemctl reset-failed 'session-*.scope'
+```
+
+Verify:
+
+```bash
+systemctl list-units --failed
+```
+
+Should report 0 failed units.
+
+## Notes
+
+- This is a known systemd behavior and not indicative of a real problem. It can be safely ignored or cleared whenever it appears.
+- If it recurs frequently, investigate what is creating short-lived SSH sessions — common culprits include monitoring agents (Netdata, Nagios), automated backup scripts, or SSH brute-force attempts.
+- The `c` in `session-c1.scope` indicates a **console/SSH session** (as opposed to graphical sessions which use different prefixes). The number increments with each new session.
+- Applies to **Fedora, Ubuntu, and any systemd-based Linux distribution**.
+
+## Related
+
+- [gitea-runner-boot-race-network-target](../gitea-runner-boot-race-network-target.md) — Another systemd race condition involving service startup ordering

05-troubleshooting/wget-url-special-characters.md

@@ -0,0 +1,100 @@
+---
+title: "wget/curl: URLs with Special Characters Fail in Bash"
+domain: troubleshooting
+category: general
+tags: [wget, curl, bash, shell, quoting, url]
+status: published
+created: 2026-04-08
+updated: 2026-04-08
+---
+# wget/curl: URLs with Special Characters Fail in Bash
+
+## Problem
+
+Downloading a URL that contains `&`, `=`, `#`, `?`, or other shell-meaningful characters fails with cryptic errors when the URL is not properly quoted:
+
+```bash
+wget -O output.mp4 https://cdn.example.com/video%20file.mp4?secure=abc123&token=xyz
+```
+
+Bash interprets `&` as a background operator, splitting the command:
+
+```
+bash: token=xyz: command not found
+```
+
+The download either fails outright or downloads only a partial/error page (e.g., 868 bytes instead of 2 GB).
+
+---
+
+## Root Cause
+
+Bash treats several URL-common characters as shell operators:
+
+| Character | Shell Meaning | URL Meaning |
+|---|---|---|
+| `&` | Run previous command in background | Query parameter separator |
+| `?` | Single-character glob wildcard | Start of query string |
+| `#` | Comment (rest of line ignored) | Fragment identifier |
+| `=` | Variable assignment (in some contexts) | Key-value separator |
+| `%` | Job control (`%1`, `%2`) | URL encoding prefix |
+| `!` | History expansion (in interactive shells) | Rarely used in URLs |
+
+If the URL is unquoted, Bash processes these characters before `wget` or `curl` ever sees them.
+
+---
+
+## Fix
+
+**Always single-quote URLs** passed to `wget` or `curl`:
+
+```bash
+wget -O '/path/to/output file.mp4' 'https://cdn.example.com/path/video%20file.mp4?secure=abc123&token=xyz'
+```
+
+Single quotes prevent **all** shell interpretation — no variable expansion, no globbing, no operator parsing. The URL reaches `wget` exactly as written.
+
+### When to use double quotes instead
+
+If the URL contains a shell variable (e.g., a token stored in `$TOKEN`), use double quotes:
+
+```bash
+wget -O output.mp4 "https://cdn.example.com/file.mp4?secure=${TOKEN}&expires=9999"
+```
+
+Double quotes allow variable expansion but still protect `&`, `?`, and `#` from shell interpretation.
+
+### Output filename quoting
+
+The `-O` filename also needs quoting if it contains spaces or special characters:
+
+```bash
+wget -O '/plex/plex/DF Direct Q+A #258.mp4' 'https://example.com/video.mp4'
+```
+
+---
+
+## Quick Reference
+
+| Scenario | Quoting |
+|---|---|
+| Static URL, no variables | Single quotes: `'https://...'` |
+| URL with shell variable | Double quotes: `"https://...${VAR}"` |
+| Output path with spaces | Single or double quotes around `-O` path |
+| URL in a script variable | Assign with double quotes: `URL="https://..."`, then `wget "$URL"` |
+
+---
+
+## Common Symptoms of Unquoted URLs
+
+- `bash: <partial-url>: command not found` — `&` split the command
+- Download completes instantly with a tiny file (error page, not the real content)
+- `wget` reports success but the file is corrupt or truncated
+- `No such file or directory` errors on URL fragments
+- History expansion errors (`!` in URL triggers `bash: !...: event not found`)
+
+---
+
+## See Also
+
+- [Bash Scripting Patterns](../01-linux/shell-scripting/bash-scripting-patterns.md) — general shell quoting and safety patterns

05-troubleshooting/yt-dlp-fedora-js-challenge.md

@@ -1,3 +1,12 @@
+---
+title: "yt-dlp YouTube JS Challenge Fix (Fedora)"
+domain: troubleshooting
+category: general
+tags: [yt-dlp, fedora, youtube, javascript, deno]
+status: published
+created: 2026-04-02
+updated: 2026-04-02
+---
 # yt-dlp YouTube JS Challenge Fix (Fedora)
 
 ## Problem
@@ -78,10 +87,6 @@ sudo pip install -U yt-dlp --break-system-packages
 
 ---
 
-## Tags
-
-#yt-dlp #fedora #youtube #plex #self-hosted
-
 ## Known Limitations
 
 ### n-Challenge Failure: "found 0 n function possibilities"
@@ -116,6 +121,26 @@ yt-dlp --list-formats --remote-components ejs:github \
   https://www.youtube.com/watch?v=VIDEO_ID
 ```
 
+### HTTP 429 Too Many Requests + Impersonation Warning
+
+Downloads or subtitle fetches fail with:
+
+```
+WARNING: The extractor specified to use impersonation for this download,
+but no impersonate target is available.
+ERROR: Unable to download video subtitles for 'en-en-US': HTTP Error 429: Too Many Requests
+```
+
+**Cause:** yt-dlp needs `curl_cffi` to impersonate a real browser's TLS fingerprint. Without it, YouTube detects the non-browser client and rate-limits with 429s. Subtitle downloads are usually the first to fail.
+
+**Fix:**
+
+```bash
+pip3 install --upgrade yt-dlp curl_cffi
+```
+
+Once `curl_cffi` is installed, yt-dlp automatically uses browser impersonation and the 429s stop. No config changes needed.
+
 ### SABR-Only Streaming Warning
 
 Some videos may show:

MajorWiki-Deploy-Status.md

@@ -2,7 +2,8 @@
 title: MajorWiki Deployment Status
 status: deployed
 project: MajorTwin
-updated: '2026-03-12'
+updated: 2026-04-07T10:48
+created: 2026-04-02T16:10
 ---
 
 # MajorWiki Deployment Status
@@ -31,8 +32,8 @@ DNS record and Caddy entry have been removed.
 
 ## Content
 
-- 42 articles across 5 domains
+- 74 articles across 5 domains
-- Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/`
+- Source of truth: `MajorVault/30-Areas/MajorWiki/`
 - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab)
 
 ## Update Workflow
@@ -40,7 +41,7 @@ DNS record and Caddy entry have been removed.
 ```bash
 # From MajorRig (majorlinux user)
 rsync -av --include="*.md" --include="*/" --exclude="*" \
-  /mnt/c/Users/majli/Documents/MajorVault/20-Projects/MajorTwin/08-Wiki/ \
+  /mnt/c/Users/majli/Documents/MajorVault/30-Areas/MajorWiki/ \
   root@majorlab:/opt/majwiki/docs/
 
 # MkDocs hot-reloads automatically — no container restart needed
@@ -71,7 +72,7 @@ Obsidian Git plugin was evaluated and dropped — too convoluted. Manual git fro
 
 ```bash
 cd ~/Documents/MajorVault
-git add 20-Projects/MajorTwin/08-Wiki/
+git add 30-Areas/MajorWiki/
 git commit -m "wiki: describe your changes"
 git push
 ```
@@ -145,3 +146,14 @@ Every time a new article is added, the following **MUST** be updated to maintain
 - `02-selfhosting/monitoring/netdata-new-server-setup.md` — full Netdata deployment guide: install via kickstart.sh, email notification config, Netdata Cloud claim
 
 **Updated:** `updated: 2026-03-18`
+
+## Session Update — 2026-04-02
+
+**Article count:** 74 (was 49)
+
+**New article this session:**
+- `02-selfhosting/security/fail2ban-wordpress-login-jail.md` — Fail2ban custom jail for WordPress login brute force (access-log-based, no plugin required)
+
+**Also today:** Major wiki audit added 8 articles from archive, fixed 67 wikilinks, added frontmatter to 43 files, moved wiki from `20-Projects/MajorTwin/08-Wiki/` to `30-Areas/MajorWiki/`. See daily note for full details.
+
+**Updated:** `updated: 2026-04-02`

README.md

@@ -1,19 +1,23 @@
+---
+created: 2026-04-06T09:52
+updated: 2026-04-13T10:16
+---
 # MajorLinux Tech Wiki — Index
 
 > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin.
 >
-**Last updated:** 2026-03-18
+**Last updated:** 2026-04-14
-**Article count:** 49
+**Article count:** 76
 
 ## Domains
 
 | Domain | Folder | Articles |
 |---|---|---|
-| 🐧 Linux & Sysadmin | `01-linux/` | 11 |
+| 🐧 Linux & Sysadmin | `01-linux/` | 12 |
-| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 11 |
+| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 22 |
-| 🔓 Open Source Tools | `03-opensource/` | 9 |
+| 🔓 Open Source Tools | `03-opensource/` | 10 |
 | 🎙️ Streaming & Podcasting | `04-streaming/` | 2 |
-| 🔧 General Troubleshooting | `05-troubleshooting/` | 16 |
+| 🔧 General Troubleshooting | `05-troubleshooting/` | 30 |
 
 ---
 
@@ -26,7 +30,7 @@
 - [Managing Linux Services with systemd](01-linux/process-management/managing-linux-services-systemd-ansible.md) — systemctl, journalctl, writing service files, Ansible service management
 
 ### Networking
-- [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys
+- [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys, Windows OpenSSH admin key auth
 
 ### Package Management
 - [Package Management Reference](01-linux/packages/package-management-reference.md) — apt, dnf, pacman side-by-side reference, Flatpak/Snap
@@ -37,6 +41,7 @@
 
 ### Storage
 - [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) — Pooling mismatched drives and adding parity on Linux
+- [mdadm — Rebuilding a RAID Array After Reinstall](01-linux/storage/mdadm-raid-rebuild.md) — reassembling and recovering mdadm arrays after OS reinstall
 
 ### Distro-Specific
 - [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments
@@ -52,24 +57,37 @@
 - [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) — hardware options, Docker install, first services, networking basics
 - [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) — when to use containers vs VMs, KVM setup, how to run both
 - [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md) — logs, inspect, exec, port conflicts, permission errors
+- [Docker Healthchecks](02-selfhosting/docker/docker-healthchecks.md) — writing and debugging HEALTHCHECK instructions in Docker containers
 
 ### Reverse Proxies
 - [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) — Caddyfile basics, automatic HTTPS, local TLS, DNS challenge
 
 ### DNS & Networking
 - [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) — installation, MagicDNS, making services accessible, subnet router, ACLs
+- [Network Overview](02-selfhosting/dns-networking/network-overview.md) — MajorsHouse network topology, Tailscale IPs, and connectivity map
 
 ### Storage & Backup
-- [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) — flags reference, remote backup, incremental with hard links, cron/systemd
+- [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) — flags reference, remote backup, incremental with hard links, Glacier Deep Archive
 
 ### Monitoring
 - [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) — tuning web_log_1m_redirects threshold for HTTPS-forcing servers
 - [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md) — preventing false alerts during nightly Nextcloud AIO container update cycles
 - [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md) — install, email notifications, and Netdata Cloud claim for Ubuntu/Debian servers
+- [Netdata + n8n Enriched Alert Emails](02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md) — rich HTML alert emails with remediation steps and wiki links via n8n
+- [Netdata SELinux AVC Denial Monitoring](02-selfhosting/monitoring/netdata-selinux-avc-chart.md) — custom Netdata chart for tracking SELinux AVC denials
 
 ### Security
-- [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban
+- [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban, SpamAssassin
 - [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) — fleet-wide automatic security updates across Ubuntu servers
+- [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) — custom filter and jail for blocking 404 scanners
+- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) — catching PHP webshell/backdoor probes that return 301 on HTTPS-redirecting servers
+- [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) — access-log-based wp-login.php brute force detection without plugins
+- [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) — resolving execmem AVC denials from Fail2ban's grep on Fedora
+- [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) — managing UFW rules, common patterns, troubleshooting
+
+### Services
+- [Updating n8n Running in Docker](02-selfhosting/services/updating-n8n-docker.md) — pinned version updates, password reset, Arcane timing gaps
+- [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md) — character limit increase, media cache management for self-hosted Mastodon
 
 ---
 
@@ -87,6 +105,7 @@
 - [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) — detachable sessions for long-running jobs over SSH
 - [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available
 - [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions
+- [Ventoy: Multi-Boot USB Tool](03-opensource/dev-tools/ventoy.md) — drop ISOs on a USB drive and boot any of them, no reflashing
 
 ### Privacy & Security
 - [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware
@@ -111,18 +130,31 @@
 - [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) — diagnosing and fixing Apache outages caused by missing firewall rules and Fail2ban self-bans
 - [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md) — diagnosing why one device stops receiving email when the mail server is healthy
 - [firewalld: Mail Ports Wiped After Reload](05-troubleshooting/networking/firewalld-mail-ports-reset.md) — recovering IMAP and webmail after firewalld reload drops all mail service rules
+- [Fail2ban & UFW Rule Bloat: 30k Rules Slowing Down a VPS](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md) — diagnosing and cleaning up massive nftables/UFW rule accumulation
+- [Tailscale SSH: Unexpected Re-Authentication Prompt](05-troubleshooting/networking/tailscale-ssh-reauth-prompt.md) — resolving unexpected re-auth prompts on Tailscale SSH connections
 - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot
+- [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md) — fixing webhook failures caused by missing proxy trust configuration
+- [Nextcloud AIO Container Unhealthy for 20 Hours](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md) — diagnosing stuck Nextcloud AIO containers after nightly update cycles
 - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber
 - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts
+- [macOS Repeating Alert Tone from Mirrored Notification](05-troubleshooting/macos-mirrored-notification-alert-loop.md) — stopping alert tone loops from mirrored iPhone notifications on Mac
 - [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning
 - [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora
 - [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail
 - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline
 - [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) — fixing act_runner crash loop on boot caused by DNS not ready at startup
+- [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) — why `/run` is tmpfs and how a reboot wipes cron heartbeat files, and where to put them instead
 - [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) — fixing thousands of AVC denials when /var/vmail has wrong SELinux context
 - [mdadm RAID Recovery After USB Hub Disconnect](05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md) — diagnosing and recovering a failed mdadm array caused by a USB hub dropout
 - [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) — fixing sshd not running after reboot due to Manual startup type
 - [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md) — keeping Ollama reachable over Tailscale by disabling macOS sleep on AC power
+- [Ansible: Vault Password File Not Found](05-troubleshooting/ansible-vault-password-file-missing.md) — fixing the missing vault_pass file error when running ansible-playbook
+- [Ansible SSH Timeout During dnf upgrade](05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md) — preventing SSH timeouts during long-running dnf upgrades on Fedora
+- [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md) — nmcli quick fix, GRUB kernel rollback, and recovery for Fedora fleet
+- [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) — blocking directory scanners and junk HTTP methods
+- [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — preventing clamscan CPU spikes with nice and ionice
+- [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md) — fixing session-cN.scope failures during login
+- [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) — fixing broken downloads caused by unquoted URLs with &, ?, # characters
 
 ---
 
@@ -130,6 +162,17 @@
 
 | Date | Article | Domain |
 |---|---|---|
+| 2026-04-13 | [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) | Troubleshooting |
+| 2026-04-09 | [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) | Self-Hosting |
+| 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting |
+| 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux |
+| 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting |
+| 2026-04-07 | [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) | Troubleshooting |
+| 2026-04-02 | [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) | Self-Hosting |
+| 2026-04-02 | [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md) | Self-Hosting |
+| 2026-04-02 | [mdadm — Rebuilding a RAID Array After Reinstall](01-linux/storage/mdadm-raid-rebuild.md) | Linux |
+| 2026-04-02 | [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md) | Troubleshooting |
+| 2026-04-02 | [Ventoy: Multi-Boot USB Tool](03-opensource/dev-tools/ventoy.md) | Open Source |
 | 2026-03-18 | [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md) | Self-Hosting |
 | 2026-03-18 | [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md) | Self-Hosting |
 | 2026-03-17 | [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md) | Troubleshooting |
@@ -169,6 +212,4 @@
 ---
 
 ## Related
-- [[MajorWiki-Deploy-Status|MajorWiki Deploy Status]] — deployment status and git workflow
+- [MajorWiki Deploy Status](MajorWiki-Deploy-Status.md) — deployment status and update workflow
-- [[01-Phases|Implementation Phases]] — Phase 9 (wiki & knowledge base)
-- [[majorlab|majorlab]] — hosting server

SUMMARY.md

@@ -1,3 +1,7 @@
+---
+created: 2026-04-02T16:03
+updated: 2026-04-13T10:16
+---
 * [Home](index.md)
 * [Linux & Sysadmin](01-linux/index.md)
     * [Linux File Permissions](01-linux/files-permissions/linux-file-permissions.md)
@@ -7,6 +11,7 @@
     * [Ansible Getting Started](01-linux/shell-scripting/ansible-getting-started.md)
     * [Bash Scripting Patterns](01-linux/shell-scripting/bash-scripting-patterns.md)
     * [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md)
+    * [mdadm — Rebuilding a RAID Array After Reinstall](01-linux/storage/mdadm-raid-rebuild.md)
     * [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md)
     * [WSL2 Instance Migration to Fedora 43](01-linux/distro-specific/wsl2-instance-migration-fedora43.md)
     * [WSL2 Training Environment Rebuild](01-linux/distro-specific/wsl2-rebuild-fedora43-training-env.md)
@@ -16,16 +21,28 @@
     * [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md)
     * [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md)
     * [Docker Healthchecks](02-selfhosting/docker/docker-healthchecks.md)
+    * [Watchtower SMTP via Localhost Postfix Relay](02-selfhosting/docker/watchtower-smtp-localhost-relay.md)
     * [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md)
     * [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md)
+    * [Network Overview](02-selfhosting/dns-networking/network-overview.md)
     * [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md)
     * [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md)
     * [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md)
     * [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md)
     * [Netdata SELinux AVC Denial Monitoring](02-selfhosting/monitoring/netdata-selinux-avc-chart.md)
     * [Netdata n8n Enriched Alert Emails](02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md)
+    * [Updating n8n Running in Docker](02-selfhosting/services/updating-n8n-docker.md)
+    * [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md)
     * [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md)
     * [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md)
+    * [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md)
+    * [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md)
+    * [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md)
+    * [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md)
+    * [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md)
+    * [Fail2ban: Enable the nginx-bad-request Jail](02-selfhosting/security/fail2ban-nginx-bad-request-jail.md)
+    * [Fail2ban Custom Jail: Apache Bad Request Detection](02-selfhosting/security/fail2ban-apache-bad-request-jail.md)
+    * [SSH Hardening Fleet-Wide with Ansible](02-selfhosting/security/ssh-hardening-ansible-fleet.md)
 * [Open Source & Alternatives](03-opensource/index.md)
     * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md)
     * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md)
@@ -34,6 +51,7 @@
     * [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md)
     * [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md)
     * [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md)
+    * [Ventoy: Multi-Boot USB Tool](03-opensource/dev-tools/ventoy.md)
     * [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md)
     * [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md)
 * [Streaming & Podcasting](04-streaming/index.md)
@@ -44,6 +62,10 @@
     * [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md)
     * [firewalld: Mail Ports Wiped After Reload](05-troubleshooting/networking/firewalld-mail-ports-reset.md)
     * [Tailscale SSH: Unexpected Re-Authentication Prompt](05-troubleshooting/networking/tailscale-ssh-reauth-prompt.md)
+    * [Fail2ban & UFW Rule Bloat Cleanup](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md)
+    * [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md)
+    * [Nextcloud AIO Unhealthy 20h After Nightly Update](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md)
+    * [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md)
     * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md)
     * [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md)
     * [Obsidian Vault Recovery — Loading Cache Hang](05-troubleshooting/obsidian-cache-hang-recovery.md)
@@ -52,9 +74,17 @@
     * [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md)
     * [MajorWiki Setup & Publishing Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md)
     * [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md)
+    * [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md)
     * [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md)
     * [mdadm RAID Recovery After USB Hub Disconnect](05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md)
     * [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md)
+    * [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md)
     * [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md)
+    * [macOS: Repeating Alert Tone from Mirrored iPhone Notification](05-troubleshooting/macos-mirrored-notification-alert-loop.md)
     * [ClamAV CPU Spike: Safe Scheduling with nice/ionice](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md)
     * [Ansible: Vault Password File Not Found](05-troubleshooting/ansible-vault-password-file-missing.md)
+    * [Ansible: ansible.cfg Ignored on WSL2 Windows Mounts](05-troubleshooting/ansible-wsl2-world-writable-mount-ignores-cfg.md)
+    * [Ansible: SSH Timeout During dnf upgrade on Fedora Hosts](05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md)
+    * [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md)
+    * [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md)
+    * [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md)

index.md

@@ -1,19 +1,23 @@
+---
+created: 2026-04-06T09:52
+updated: 2026-04-13T10:16
+---
 # MajorLinux Tech Wiki — Index
 
 > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin.
 >
-> **Last updated:** 2026-03-27
+> **Last updated:** 2026-04-14
-> **Article count:** 53
+> **Article count:** 76
 
 ## Domains
 
 | Domain | Folder | Articles |
 |---|---|---|
-| 🐧 Linux & Sysadmin | `01-linux/` | 11 |
+| 🐧 Linux & Sysadmin | `01-linux/` | 12 |
-| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 11 |
+| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 22 |
-| 🔓 Open Source Tools | `03-opensource/` | 9 |
+| 🔓 Open Source Tools | `03-opensource/` | 10 |
 | 🎙️ Streaming & Podcasting | `04-streaming/` | 2 |
-| 🔧 General Troubleshooting | `05-troubleshooting/` | 17 |
+| 🔧 General Troubleshooting | `05-troubleshooting/` | 30 |
 
 
 ---
@@ -27,7 +31,7 @@
 - [Managing Linux Services with systemd](01-linux/process-management/managing-linux-services-systemd-ansible.md) — systemctl, journalctl, writing service files, Ansible service management
 
 ### Networking
-- [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys
+- [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) — key generation, ssh-copy-id, ~/.ssh/config, managing multiple keys, Windows OpenSSH admin key auth
 
 ### Package Management
 - [Package Management Reference](01-linux/packages/package-management-reference.md) — apt, dnf, pacman side-by-side reference, Flatpak/Snap
@@ -38,6 +42,7 @@
 
 ### Storage
 - [SnapRAID & MergerFS Storage Setup](01-linux/storage/snapraid-mergerfs-setup.md) — Pooling mismatched drives and adding parity on Linux
+- [mdadm — Rebuilding a RAID Array After Reinstall](01-linux/storage/mdadm-raid-rebuild.md) — reassembling and recovering mdadm arrays after OS reinstall
 
 ### Distro-Specific
 - [Linux Distro Guide for Beginners](01-linux/distro-specific/linux-distro-guide-beginners.md) — Ubuntu recommendation, distro comparison, desktop environments
@@ -53,12 +58,14 @@
 - [Self-Hosting Starter Guide](02-selfhosting/docker/self-hosting-starter-guide.md) — hardware options, Docker install, first services, networking basics
 - [Docker vs VMs for the Homelab](02-selfhosting/docker/docker-vs-vms-homelab.md) — when to use containers vs VMs, KVM setup, how to run both
 - [Debugging Broken Docker Containers](02-selfhosting/docker/debugging-broken-docker-containers.md) — logs, inspect, exec, port conflicts, permission errors
+- [Docker Healthchecks](02-selfhosting/docker/docker-healthchecks.md) — writing and debugging HEALTHCHECK instructions in Docker containers
 
 ### Reverse Proxies
 - [Setting Up Caddy as a Reverse Proxy](02-selfhosting/reverse-proxy/setting-up-caddy-reverse-proxy.md) — Caddyfile basics, automatic HTTPS, local TLS, DNS challenge
 
 ### DNS & Networking
 - [Tailscale for Homelab Remote Access](02-selfhosting/dns-networking/tailscale-homelab-remote-access.md) — installation, MagicDNS, making services accessible, subnet router, ACLs
+- [Network Overview](02-selfhosting/dns-networking/network-overview.md) — MajorsHouse network topology, Tailscale IPs, and connectivity map
 
 ### Storage & Backup
 - [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) — flags reference, remote backup, incremental with hard links, cron/systemd
@@ -67,10 +74,21 @@
 - [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) — tuning web_log_1m_redirects threshold for HTTPS-forcing servers
 - [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md) — preventing false alerts during nightly Nextcloud AIO container update cycles
 - [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md) — install, email notifications, and Netdata Cloud claim for Ubuntu/Debian servers
+- [Netdata + n8n Enriched Alert Emails](02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md) — rich HTML alert emails with remediation steps and wiki links via n8n
+- [Netdata SELinux AVC Denial Monitoring](02-selfhosting/monitoring/netdata-selinux-avc-chart.md) — custom Netdata chart for tracking SELinux AVC denials
 
 ### Security
-- [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban
+- [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban, SpamAssassin
 - [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) — fleet-wide automatic security updates across Ubuntu servers
+- [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) — custom filter and jail for blocking 404 scanners
+- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) — catching PHP webshell/backdoor probes that return 301 on HTTPS-redirecting servers
+- [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) — access-log-based wp-login.php brute force detection without plugins
+- [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) — resolving execmem AVC denials from Fail2ban's grep on Fedora
+- [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) — managing UFW rules, common patterns, troubleshooting
+
+### Services
+- [Updating n8n Running in Docker](02-selfhosting/services/updating-n8n-docker.md) — pinned version updates, password reset, Arcane timing gaps
+- [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md) — character limit increase, media cache management for self-hosted Mastodon
 
 ---
 
@@ -88,6 +106,7 @@
 - [tmux: Persistent Terminal Sessions](03-opensource/dev-tools/tmux.md) — detachable sessions for long-running jobs over SSH
 - [screen: Simple Persistent Sessions](03-opensource/dev-tools/screen.md) — lightweight terminal multiplexer, universally available
 - [rsync: Fast, Resumable File Transfers](03-opensource/dev-tools/rsync.md) — incremental file sync locally and over SSH, survives interruptions
+- [Ventoy: Multi-Boot USB Tool](03-opensource/dev-tools/ventoy.md) — drop ISOs on a USB drive and boot any of them, no reflashing
 
 ### Privacy & Security
 - [Vaultwarden: Self-Hosted Password Manager](03-opensource/privacy-security/vaultwarden.md) — Bitwarden-compatible server in a single Docker container, passwords stay on your hardware
@@ -112,19 +131,33 @@
 - [Apache Outage: Fail2ban Self-Ban + Missing iptables Rules](05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md) — diagnosing and fixing Apache outages caused by missing firewall rules and Fail2ban self-bans
 - [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md) — diagnosing why one device stops receiving email when the mail server is healthy
 - [firewalld: Mail Ports Wiped After Reload](05-troubleshooting/networking/firewalld-mail-ports-reset.md) — recovering IMAP and webmail after firewalld reload drops all mail service rules
+- [Fail2ban & UFW Rule Bloat: 30k Rules Slowing Down a VPS](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md) — diagnosing and cleaning up massive nftables/UFW rule accumulation
+- [Tailscale SSH: Unexpected Re-Authentication Prompt](05-troubleshooting/networking/tailscale-ssh-reauth-prompt.md) — resolving unexpected re-auth prompts on Tailscale SSH connections
 - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) — fixing docker.socket, SELinux port blocks, and httpd_can_network_connect after reboot
+- [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md) — fixing webhook failures caused by missing proxy trust configuration
+- [Nextcloud AIO Container Unhealthy for 20 Hours](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md) — diagnosing stuck Nextcloud AIO containers after nightly update cycles
 - [ISP SNI Filtering with Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) — troubleshooting why wiki.majorshouse.com was blocked by Google Fiber
 - [Obsidian Cache Hang Recovery](05-troubleshooting/obsidian-cache-hang-recovery.md) — resolving "Loading cache" hang in Obsidian by cleaning Electron app data and ML artifacts
+- [macOS Repeating Alert Tone from Mirrored Notification](05-troubleshooting/macos-mirrored-notification-alert-loop.md) — stopping alert tone loops from mirrored iPhone notifications on Mac
 - [Qwen2.5-14B OOM on RTX 3080 Ti (12GB)](05-troubleshooting/gpu-display/qwen-14b-oom-3080ti.md) — fixes and alternatives when hitting VRAM limits during fine-tuning
 - [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora
 - [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail
 - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline
 - [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) — fixing act_runner crash loop on boot caused by DNS not ready at startup
+- [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) — why `/run` is tmpfs and how a reboot wipes cron heartbeat files, and where to put them instead
 - [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) — fixing thousands of AVC denials when /var/vmail has wrong SELinux context
 - [mdadm RAID Recovery After USB Hub Disconnect](05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md) — diagnosing and recovering a failed mdadm array caused by a USB hub dropout
 - [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) — fixing sshd not running after reboot due to Manual startup type
+- [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) — fixing remote SSH command failures when wsl.exe is the default shell
 - [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md) — keeping Ollama reachable over Tailscale by disabling macOS sleep on AC power
 - [Ansible: Vault Password File Not Found](05-troubleshooting/ansible-vault-password-file-missing.md) — fixing the missing vault_pass file error when running ansible-playbook
+- [Ansible: ansible.cfg Ignored on WSL2 Windows Mounts](05-troubleshooting/ansible-wsl2-world-writable-mount-ignores-cfg.md) — fixing silent config ignore due to world-writable /mnt/d/ permissions
+- [Ansible SSH Timeout During dnf upgrade](05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md) — preventing SSH timeouts during long-running dnf upgrades on Fedora
+- [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md) — nmcli quick fix, GRUB kernel rollback, and recovery for Fedora fleet
+- [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) — blocking directory scanners and junk HTTP methods
+- [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) — preventing clamscan CPU spikes with nice and ionice
+- [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md) — fixing session-cN.scope failures during login
+- [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) — fixing broken downloads caused by unquoted URLs with &, ?, # characters
 
 
 ---
@@ -133,10 +166,23 @@
 
 | Date | Article | Domain |
 |---|---|---|
-<<<<<<< HEAD
+| 2026-04-13 | [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) | Troubleshooting |
+| 2026-04-09 | [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) | Self-Hosting |
+| 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting |
+| 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux |
+| 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting |
+| 2026-04-07 | [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md) | Troubleshooting |
+| 2026-04-03 | [Ansible: ansible.cfg Ignored on WSL2 Windows Mounts](05-troubleshooting/ansible-wsl2-world-writable-mount-ignores-cfg.md) | Troubleshooting |
+| 2026-04-02 | [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) | Self-Hosting |
+| 2026-04-02 | [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md) | Self-Hosting |
+| 2026-04-02 | [mdadm — Rebuilding a RAID Array After Reinstall](01-linux/storage/mdadm-raid-rebuild.md) | Linux |
+| 2026-04-02 | [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md) | Troubleshooting |
+| 2026-04-02 | [Ventoy: Multi-Boot USB Tool](03-opensource/dev-tools/ventoy.md) | Open Source |
+| 2026-04-02 | [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) (updated — Glacier Deep Archive) | Self-Hosting |
+| 2026-04-02 | [yt-dlp: Video Downloading](03-opensource/media-creative/yt-dlp.md) (updated — subtitles, temp fix) | Open Source |
+| 2026-04-02 | [OBS Studio Setup & Encoding](04-streaming/obs/obs-studio-setup-encoding.md) (updated — captions plugin, VLC capture) | Streaming |
+| 2026-04-02 | [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) (updated — SpamAssassin) | Self-Hosting |
 | 2026-03-23 | [Ansible: Vault Password File Not Found](05-troubleshooting/ansible-vault-password-file-missing.md) | Troubleshooting |
-=======
->>>>>>> 335c4b57f20799b3a968460f4f6aa17a8b706fdc
 | 2026-03-18 | [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md) | Self-Hosting |
 | 2026-03-18 | [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md) | Self-Hosting |
 | 2026-03-17 | [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md) | Troubleshooting |
@@ -175,6 +221,4 @@
 ---
 
 ## Related
-- [[MajorWiki-Deploy-Status|MajorWiki Deploy Status]] — deployment status and update workflow
+- [MajorWiki Deploy Status](MajorWiki-Deploy-Status.md) — deployment status and update workflow
-- [[01-Phases|Implementation Phases]] — Phase 9 (wiki & knowledge base)
-- [[majorlab|majorlab]] — hosting server (notes.majorshouse.com)