merge: resolve conflicts, keep new IMAP self-ban article

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

05-troubleshooting/networking/fail2ban-self-ban-apache-outage.md

@@ -0,0 +1,186 @@
+# Apache Outage: Fail2ban Self-Ban + Missing iptables Rules
+
+## 🛑 Problem
+
+A web server running Apache2 becomes completely unreachable (`ERR_CONNECTION_TIMED_OUT`) despite Apache running normally. SSH access via Tailscale is unaffected.
+
+---
+
+## 🔍 Diagnosis
+
+### Step 1 — Confirm Apache is running
+
+```bash
+sudo systemctl status apache2
+```
+
+If Apache is `active (running)`, the problem is at the firewall layer, not the application.
+
+---
+
+### Step 2 — Test the public IP directly
+
+```bash
+curl -I --max-time 5 http://<PUBLIC_IP>
+```
+
+A **timeout** means traffic is being dropped by the firewall. A **connection refused** means Apache is down.
+
+---
+
+### Step 3 — Check the iptables INPUT chain
+
+```bash
+sudo iptables -L INPUT -n -v
+```
+
+Look for ACCEPT rules on ports 80 and 443. If they're missing and the chain policy is `DROP`, HTTP/HTTPS traffic is being silently dropped.
+
+**Example of broken state:**
+```
+Chain INPUT (policy DROP)
+  ACCEPT  tcp  --  lo       *    ...   # loopback only
+  ACCEPT  tcp  --  tailscale0 *  ...   tcp dpt:22
+  # no rules for port 80 or 443
+```
+
+---
+
+### Step 4 — Check the nftables ruleset for Fail2ban
+
+```bash
+sudo nft list tables
+```
+
+Look for `table inet f2b-table` — this is Fail2ban's nftables table. It operates at **priority `filter - 1`**, meaning it is evaluated *before* the main iptables INPUT chain.
+
+```bash
+sudo nft list ruleset | grep -A 10 'f2b-table'
+```
+
+Fail2ban rejects banned IPs with rules like:
+```
+tcp dport { 80, 443 } ip saddr @addr-set-wordpress-hard reject with icmp port-unreachable
+```
+
+A banned admin IP will be rejected here regardless of any ACCEPT rules downstream.
+
+---
+
+### Step 5 — Check if your IP is banned
+
+```bash
+for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
+  echo "=== $jail ==="; sudo fail2ban-client get $jail banip | tr ',' '\n' | grep <YOUR_IP>
+done
+```
+
+---
+
+## ✅ Solution
+
+### Fix 1 — Add missing iptables ACCEPT rules for HTTP/HTTPS
+
+If ports 80/443 are absent from the INPUT chain:
+
+```bash
+sudo iptables -I INPUT -i eth0 -p tcp --dport 80 -j ACCEPT
+sudo iptables -I INPUT -i eth0 -p tcp --dport 443 -j ACCEPT
+```
+
+Persist the rules:
+
+```bash
+sudo netfilter-persistent save
+```
+
+If `netfilter-persistent` is not installed:
+
+```bash
+sudo apt install -y iptables-persistent
+sudo netfilter-persistent save
+```
+
+---
+
+### Fix 2 — Unban your IP from all Fail2ban jails
+
+```bash
+for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
+  sudo fail2ban-client set $jail unbanip <YOUR_IP> 2>/dev/null && echo "Unbanned from $jail"
+done
+```
+
+---
+
+### Fix 3 — Add your IP to Fail2ban's ignore list
+
+Edit `/etc/fail2ban/jail.local`:
+
+```bash
+sudo nano /etc/fail2ban/jail.local
+```
+
+Add or update the `[DEFAULT]` section:
+
+```ini
+[DEFAULT]
+ignoreip = 127.0.0.1/8 ::1 <YOUR_IP>
+```
+
+Restart Fail2ban:
+
+```bash
+sudo systemctl restart fail2ban
+```
+
+---
+
+## 🔁 Why This Happens
+
+| Issue | Root Cause |
+|---|---|
+| Missing port 80/443 rules | iptables INPUT chain left incomplete after a manual firewall rework (e.g., SSH lockdown) |
+| Still blocked after adding iptables rules | Fail2ban uses a separate nftables table at higher priority — iptables ACCEPT rules are never reached for banned IPs |
+| Admin IP gets banned | Automated WordPress/Apache probes trigger Fail2ban jails against the admin's own IP |
+
+---
+
+## ⚠️ Key Architecture Note
+
+On servers running both iptables and Fail2ban, the evaluation order is:
+
+1. **`inet f2b-table`** (nftables, priority `filter - 1`) — Fail2ban ban sets; evaluated first
+2. **`ip filter` INPUT chain** (iptables/nftables, policy DROP) — explicit ACCEPT rules
+3. **UFW chains** — IP-specific rules; evaluated last
+
+A banned IP is stopped at step 1 and never reaches the ACCEPT rules in step 2. Always check Fail2ban *after* confirming iptables looks correct.
+
+---
+
+## 🔎 Quick Diagnostic Commands
+
+```bash
+# Check Apache
+sudo systemctl status apache2
+
+# Test public connectivity
+curl -I --max-time 5 http://<PUBLIC_IP>
+
+# Check iptables INPUT chain
+sudo iptables -L INPUT -n -v
+
+# List nftables tables (look for inet f2b-table)
+sudo nft list tables
+
+# Check Fail2ban jail status
+sudo fail2ban-client status
+
+# Check a specific jail's banned IPs
+sudo fail2ban-client status wordpress-hard
+
+# Unban an IP from all jails
+for jail in $(sudo fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
+  sudo fail2ban-client set $jail unbanip <YOUR_IP> 2>/dev/null && echo "Unbanned from $jail"
+done
+```