diff --git a/02-selfhosting/security/fail2ban-apache-404-scanner-jail.md b/02-selfhosting/security/fail2ban-apache-404-scanner-jail.md new file mode 100644 index 0000000..7ae7a27 --- /dev/null +++ b/02-selfhosting/security/fail2ban-apache-404-scanner-jail.md @@ -0,0 +1,110 @@ +# Fail2ban Custom Jail: Apache 404 Scanner Detection + +## The Problem + +Automated vulnerability scanners probe web servers by requesting dozens of common config file paths — `.env`, `env.php`, `next.config.js`, `nuxt.config.ts`, etc. — in rapid succession. These all return **404 Not Found**, which is correct behavior from Apache. + +However, the built-in Fail2ban jails (`apache-noscript`, `apache-botsearch`) don't catch these because they parse the **error log**, not the **access log**. If Apache doesn't write a corresponding "File does not exist" entry to the error log for every 404, the scanner slips through undetected. + +This also triggers false alerts in monitoring tools like **Netdata**, which sees the success ratio drop (e.g., `web_log_1m_successful` goes CRITICAL at 2.83%) because 404s aren't counted as successful responses. + +## The Solution + +Create a custom Fail2ban filter that reads the **access log** and matches 404 responses directly. + +### Step 1 — Create the filter + +Create `/etc/fail2ban/filter.d/apache-404scan.conf`: + +```ini +# Fail2Ban filter to catch rapid 404 scanning in Apache access logs +# Targets vulnerability scanners probing for .env, config files, etc. + +[Definition] + +# Match 404 responses in combined/common access log format +failregex = ^ -.*"(GET|POST|HEAD|PUT|DELETE|OPTIONS|PATCH) .+" 404 \d+ + +ignoreregex = ^ -.*(robots\.txt|favicon\.ico|apple-touch-icon) + +datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z +``` + +### Step 2 — Add the jail + +Add to `/etc/fail2ban/jail.local`: + +```ini +[apache-404scan] +enabled = true +port = http,https +filter = apache-404scan +logpath = /var/log/apache2/access.log +maxretry = 10 +findtime = 1m +bantime = 24h +``` + +**10 hits in 1 minute** is aggressive enough to catch scanners (which fire 30–50+ requests in seconds) while avoiding false positives from a legitimate user hitting a few broken links. + +### Step 3 — Test the regex + +```bash +fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404scan.conf +``` + +You should see matches. In a real-world test against a server under active scanning, this matched **2831 out of 8901** access log lines. + +### Step 4 — Reload Fail2ban + +```bash +systemctl restart fail2ban +fail2ban-client status apache-404scan +``` + +## Why Default Jails Miss This + +| Jail | Log Source | What It Matches | Why It Misses | +|---|---|---|---| +| `apache-noscript` | error log | "script not found or unable to stat" | Only matches script-type files (.php, .asp, .exe, .pl) | +| `apache-botsearch` | error log | "File does not exist" for specific paths | Requires Apache to write error log entries for 404s | +| **`apache-404scan`** | **access log** | **Any 404 response** | **Catches everything** | + +The key insight: URL-encoded probes like `/%2f%2eenv%2econfig` that return 404 in the access log may not generate error log entries at all, making them invisible to the default filters. + +## Pair With Recidive + +If you have the `recidive` jail enabled, repeat offenders get permanently banned: + +```ini +[recidive] +enabled = true +bantime = -1 +findtime = 86400 +maxretry = 3 +``` + +Three 24-hour bans within a day = permanent firewall block. + +## Quick Diagnostic Commands + +```bash +# Test filter against current access log +fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-404scan.conf + +# Check jail status and banned IPs +fail2ban-client status apache-404scan + +# Watch bans in real time +tail -f /var/log/fail2ban.log | grep apache-404scan + +# Count 404s in today's log +grep '" 404 ' /var/log/apache2/access.log | wc -l +``` + +## Key Notes + +- The `ignoreregex` excludes `robots.txt`, `favicon.ico`, and `apple-touch-icon` — these are commonly requested and produce harmless 404s. +- Make sure your Tailscale subnet (`100.64.0.0/10`) is in the `ignoreip` list under `[DEFAULT]` so you don't ban your own monitoring or uptime checks. +- This filter works with both Apache **combined** and **common** log formats. +- Complements the existing `apache-dirscan` jail (which catches error-log-based directory enumeration). Use both for full coverage. diff --git a/SUMMARY.md b/SUMMARY.md index b1e5991..5dd62da 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -26,6 +26,7 @@ * [Netdata n8n Enriched Alert Emails](02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md) * [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) * [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) + * [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) * [Open Source & Alternatives](03-opensource/index.md) * [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) * [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md)