wiki: add fail2ban jail for Apache PHP webshell probes

Documents the 2026-04-09 scanner incident where 301-redirected PHP probes
bypassed the existing apache-404scan jail, leaving the scanner unbanned
and firing Netdata web_log_1m_redirects alerts. New jail catches 301/302/
403/404 PHP responses while excluding legitimate WordPress endpoints.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

02-selfhosting/security/fail2ban-apache-php-probe-jail.md

@@ -0,0 +1,146 @@
+---
+title: "Fail2ban Custom Jail: Apache PHP Webshell Probe Detection"
+domain: selfhosting
+category: security
+tags:
+  - fail2ban
+  - apache
+  - security
+  - php
+  - webshell
+  - scanner
+status: published
+created: 2026-04-09
+updated: 2026-04-13T10:15
+---
+# Fail2ban Custom Jail: Apache PHP Webshell Probe Detection
+
+## The Problem
+
+Automated scanners flood web servers with rapid-fire requests for non-existent `.php` files — `bless.php`, `alfa.php`, `lock360.php`, `about.php`, `cgi-bin/bypass.php`, and hundreds of others. These are classic **webshell/backdoor probes** looking for compromised PHP files left behind by prior attackers.
+
+On servers that force HTTPS (or have HTTP→HTTPS redirects in place), these probes often return **301 Moved Permanently** instead of 404. That causes three problems:
+
+1. **The `apache-404scan` jail misses them** — it only matches 404 responses
+2. **Netdata fires false `web_log_1m_redirects` alerts** — the redirect ratio spikes to 96%+ during scans
+3. **The scanner is never banned**, and will return repeatedly
+
+This was the exact trigger for the 2026-04-09 `[MajorLinux] Web Log Alert` incident where `45.86.202.224` sent 202 PHP probe requests in a few minutes, all returning 301.
+
+## The Solution
+
+Create a custom Fail2ban filter that matches **any `.php` request returning a redirect, forbidden, or not-found response** — while excluding legitimate WordPress PHP endpoints.
+
+### Step 1 — Create the filter
+
+Create `/etc/fail2ban/filter.d/apache-php-probe.conf`:
+
+```ini
+# Fail2Ban filter to catch PHP file probing (webshell/backdoor scanners)
+# These requests hit non-existent .php files and get 301/302/403/404 responses
+
+[Definition]
+
+failregex = ^<HOST> -.*"(GET|POST|HEAD) /[^ ]*\.php[^ ]* HTTP/[0-9.]+" (301|302|403|404) \d+
+
+ignoreregex = ^<HOST> -.*(wp-cron\.php|xmlrpc\.php|wp-login\.php|wp-admin|index\.php|wp-comments-post\.php)
+
+datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
+```
+
+**Why the ignoreregex matters:** Legitimate WordPress traffic hits `wp-cron.php`, `xmlrpc.php` (often 403-blocked on hardened sites), `wp-login.php`, and `index.php` constantly. Without exclusions the jail would ban your own WordPress admins. Note that `wp-login.php` brute force is caught separately by the `wordpress` jail.
+
+### Step 2 — Add the jail
+
+Add to `/etc/fail2ban/jail.local`:
+
+```ini
+[apache-php-probe]
+enabled = true
+port = http,https
+filter = apache-php-probe
+logpath = /var/log/apache2/access.log
+maxretry = 5
+findtime = 1m
+bantime = 48h
+```
+
+**5 hits in 1 minute** is tight — scanners fire 20–200 PHP probes in seconds, while a real user hitting one broken PHP link won't trip the threshold. The 48-hour bantime is longer than `apache-404scan`'s 24h because PHP webshell scanning is a stronger signal of malicious intent.
+
+### Step 3 — Test the regex
+
+```bash
+fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-php-probe.conf
+```
+
+Verify it matches the scanner requests and does **not** match legitimate WordPress traffic.
+
+### Step 4 — Reload Fail2ban
+
+```bash
+systemctl restart fail2ban
+fail2ban-client status apache-php-probe
+```
+
+## Why This Complements `apache-404scan`
+
+| Jail | Catches | Misses |
+|---|---|---|
+| `apache-404scan` | Any 404 (config file probes, `.env`, random paths) | PHP probes redirected to HTTPS (301) |
+| **`apache-php-probe`** | **PHP webshell probes (301/302/403/404)** | Non-`.php` probes |
+
+Running both jails together covers:
+- **HTTP→HTTPS redirected PHP probes** (301 responses)
+- **Directly-served PHP probes** (404 responses)
+- **Blocked PHP paths** like `xmlrpc.php` in non-WP contexts (403 responses)
+
+## Pair With Recidive
+
+The `recidive` jail catches repeat offenders across all jails:
+
+```ini
+[recidive]
+enabled = true
+bantime = -1
+findtime = 86400
+maxretry = 3
+```
+
+A scanner that trips `apache-php-probe` three times in 24 hours gets a **permanent** firewall-level ban.
+
+## Manual IP Blocking via UFW
+
+For known scanners you want to block immediately without waiting for the jail to trip, use UFW:
+
+```bash
+# Insert at top of rule list (priority over Apache ALLOW rules)
+ufw insert 1 deny from <IP> to any comment "PHP webshell scanner YYYY-MM-DD"
+```
+
+This bypasses fail2ban entirely and is useful for:
+- Scanners you spot in logs after the fact
+- Known-malicious subnets from threat intel
+- Entire CIDR blocks (`ufw insert 1 deny from 45.86.202.0/24`)
+
+## Quick Diagnostic Commands
+
+```bash
+# Count recent PHP probes returning 301/403/404
+awk '/09\/Apr\/2026:18:/ && /\.php/ && ($9==301 || $9==403 || $9==404)' /var/log/apache2/access.log | wc -l
+
+# Top probed PHP filenames (useful for writing additional ignoreregex)
+grep '\.php' /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
+
+# Top scanner IPs by PHP probe count
+grep '\.php' /var/log/apache2/access.log | awk '$9 ~ /^(301|403|404)$/ {print $1}' | sort | uniq -c | sort -rn | head -10
+
+# Watch bans in real time
+tail -f /var/log/fail2ban.log | grep apache-php-probe
+```
+
+## Key Notes
+
+- **This jail only makes sense on servers that redirect HTTP→HTTPS.** On plain-HTTPS-only servers, PHP probes return 404 and `apache-404scan` already catches them.
+- **Add your own WordPress plugin paths to `ignoreregex`** if you use non-standard endpoints (e.g., custom admin URLs, REST API `.php` handlers).
+- **This filter pairs naturally with Netdata `web_log_1m_redirects` alerts** — during a scan, Netdata fires first (threshold crossed), then fail2ban bans the IP within seconds.
+- Also see: [Fail2ban Custom Jail: Apache 404 Scanner Detection](fail2ban-apache-404-scanner-jail.md) for the sibling 404-based filter.