From c0837b7e89244e63021aef3d3fbbc4f6c712ce74 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Mon, 13 Apr 2026 10:17:24 -0400 Subject: [PATCH] wiki: add fail2ban jail for Apache PHP webshell probes Documents the 2026-04-09 scanner incident where 301-redirected PHP probes bypassed the existing apache-404scan jail, leaving the scanner unbanned and firing Netdata web_log_1m_redirects alerts. New jail catches 301/302/ 403/404 PHP responses while excluding legitimate WordPress endpoints. Co-Authored-By: Claude Opus 4.6 (1M context) --- 02-selfhosting/index.md | 5 + .../fail2ban-apache-php-probe-jail.md | 146 ++++++++++++++++++ README.md | 4 +- SUMMARY.md | 3 +- index.md | 4 +- 5 files changed, 159 insertions(+), 3 deletions(-) create mode 100644 02-selfhosting/security/fail2ban-apache-php-probe-jail.md diff --git a/02-selfhosting/index.md b/02-selfhosting/index.md index 61751ec..40a08c3 100644 --- a/02-selfhosting/index.md +++ b/02-selfhosting/index.md @@ -1,3 +1,7 @@ +--- +created: 2026-04-13T10:15 +updated: 2026-04-13T10:15 +--- # 🏠 Self-Hosting & Homelab Guides for running your own services at home, including Docker, reverse proxies, DNS, storage, monitoring, and security. @@ -31,6 +35,7 @@ Guides for running your own services at home, including Docker, reverse proxies, - [Linux Server Hardening Checklist](security/linux-server-hardening-checklist.md) - [Standardizing unattended-upgrades with Ansible](security/ansible-unattended-upgrades-fleet.md) - [Fail2ban Custom Jail: Apache 404 Scanner Detection](security/fail2ban-apache-404-scanner-jail.md) +- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](security/fail2ban-apache-php-probe-jail.md) - [Fail2ban Custom Jail: WordPress Login Brute Force](security/fail2ban-wordpress-login-jail.md) - [SELinux: Fixing Fail2ban grep execmem Denial](security/selinux-fail2ban-execmem-fix.md) - [UFW Firewall Management](security/ufw-firewall-management.md) diff --git a/02-selfhosting/security/fail2ban-apache-php-probe-jail.md b/02-selfhosting/security/fail2ban-apache-php-probe-jail.md new file mode 100644 index 0000000..af65c66 --- /dev/null +++ b/02-selfhosting/security/fail2ban-apache-php-probe-jail.md @@ -0,0 +1,146 @@ +--- +title: "Fail2ban Custom Jail: Apache PHP Webshell Probe Detection" +domain: selfhosting +category: security +tags: + - fail2ban + - apache + - security + - php + - webshell + - scanner +status: published +created: 2026-04-09 +updated: 2026-04-13T10:15 +--- +# Fail2ban Custom Jail: Apache PHP Webshell Probe Detection + +## The Problem + +Automated scanners flood web servers with rapid-fire requests for non-existent `.php` files — `bless.php`, `alfa.php`, `lock360.php`, `about.php`, `cgi-bin/bypass.php`, and hundreds of others. These are classic **webshell/backdoor probes** looking for compromised PHP files left behind by prior attackers. + +On servers that force HTTPS (or have HTTP→HTTPS redirects in place), these probes often return **301 Moved Permanently** instead of 404. That causes three problems: + +1. **The `apache-404scan` jail misses them** — it only matches 404 responses +2. **Netdata fires false `web_log_1m_redirects` alerts** — the redirect ratio spikes to 96%+ during scans +3. **The scanner is never banned**, and will return repeatedly + +This was the exact trigger for the 2026-04-09 `[MajorLinux] Web Log Alert` incident where `45.86.202.224` sent 202 PHP probe requests in a few minutes, all returning 301. + +## The Solution + +Create a custom Fail2ban filter that matches **any `.php` request returning a redirect, forbidden, or not-found response** — while excluding legitimate WordPress PHP endpoints. + +### Step 1 — Create the filter + +Create `/etc/fail2ban/filter.d/apache-php-probe.conf`: + +```ini +# Fail2Ban filter to catch PHP file probing (webshell/backdoor scanners) +# These requests hit non-existent .php files and get 301/302/403/404 responses + +[Definition] + +failregex = ^ -.*"(GET|POST|HEAD) /[^ ]*\.php[^ ]* HTTP/[0-9.]+" (301|302|403|404) \d+ + +ignoreregex = ^ -.*(wp-cron\.php|xmlrpc\.php|wp-login\.php|wp-admin|index\.php|wp-comments-post\.php) + +datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z +``` + +**Why the ignoreregex matters:** Legitimate WordPress traffic hits `wp-cron.php`, `xmlrpc.php` (often 403-blocked on hardened sites), `wp-login.php`, and `index.php` constantly. Without exclusions the jail would ban your own WordPress admins. Note that `wp-login.php` brute force is caught separately by the `wordpress` jail. + +### Step 2 — Add the jail + +Add to `/etc/fail2ban/jail.local`: + +```ini +[apache-php-probe] +enabled = true +port = http,https +filter = apache-php-probe +logpath = /var/log/apache2/access.log +maxretry = 5 +findtime = 1m +bantime = 48h +``` + +**5 hits in 1 minute** is tight — scanners fire 20–200 PHP probes in seconds, while a real user hitting one broken PHP link won't trip the threshold. The 48-hour bantime is longer than `apache-404scan`'s 24h because PHP webshell scanning is a stronger signal of malicious intent. + +### Step 3 — Test the regex + +```bash +fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-php-probe.conf +``` + +Verify it matches the scanner requests and does **not** match legitimate WordPress traffic. + +### Step 4 — Reload Fail2ban + +```bash +systemctl restart fail2ban +fail2ban-client status apache-php-probe +``` + +## Why This Complements `apache-404scan` + +| Jail | Catches | Misses | +|---|---|---| +| `apache-404scan` | Any 404 (config file probes, `.env`, random paths) | PHP probes redirected to HTTPS (301) | +| **`apache-php-probe`** | **PHP webshell probes (301/302/403/404)** | Non-`.php` probes | + +Running both jails together covers: +- **HTTP→HTTPS redirected PHP probes** (301 responses) +- **Directly-served PHP probes** (404 responses) +- **Blocked PHP paths** like `xmlrpc.php` in non-WP contexts (403 responses) + +## Pair With Recidive + +The `recidive` jail catches repeat offenders across all jails: + +```ini +[recidive] +enabled = true +bantime = -1 +findtime = 86400 +maxretry = 3 +``` + +A scanner that trips `apache-php-probe` three times in 24 hours gets a **permanent** firewall-level ban. + +## Manual IP Blocking via UFW + +For known scanners you want to block immediately without waiting for the jail to trip, use UFW: + +```bash +# Insert at top of rule list (priority over Apache ALLOW rules) +ufw insert 1 deny from to any comment "PHP webshell scanner YYYY-MM-DD" +``` + +This bypasses fail2ban entirely and is useful for: +- Scanners you spot in logs after the fact +- Known-malicious subnets from threat intel +- Entire CIDR blocks (`ufw insert 1 deny from 45.86.202.0/24`) + +## Quick Diagnostic Commands + +```bash +# Count recent PHP probes returning 301/403/404 +awk '/09\/Apr\/2026:18:/ && /\.php/ && ($9==301 || $9==403 || $9==404)' /var/log/apache2/access.log | wc -l + +# Top probed PHP filenames (useful for writing additional ignoreregex) +grep '\.php' /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20 + +# Top scanner IPs by PHP probe count +grep '\.php' /var/log/apache2/access.log | awk '$9 ~ /^(301|403|404)$/ {print $1}' | sort | uniq -c | sort -rn | head -10 + +# Watch bans in real time +tail -f /var/log/fail2ban.log | grep apache-php-probe +``` + +## Key Notes + +- **This jail only makes sense on servers that redirect HTTP→HTTPS.** On plain-HTTPS-only servers, PHP probes return 404 and `apache-404scan` already catches them. +- **Add your own WordPress plugin paths to `ignoreregex`** if you use non-standard endpoints (e.g., custom admin URLs, REST API `.php` handlers). +- **This filter pairs naturally with Netdata `web_log_1m_redirects` alerts** — during a scan, Netdata fires first (threshold crossed), then fail2ban bans the IP within seconds. +- Also see: [Fail2ban Custom Jail: Apache 404 Scanner Detection](fail2ban-apache-404-scanner-jail.md) for the sibling 404-based filter. diff --git a/README.md b/README.md index 4a46156..754731f 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ --- created: 2026-04-06T09:52 -updated: 2026-04-13T10:10 +updated: 2026-04-13T10:16 --- # MajorLinux Tech Wiki — Index @@ -80,6 +80,7 @@ updated: 2026-04-13T10:10 - [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban, SpamAssassin - [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) — fleet-wide automatic security updates across Ubuntu servers - [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) — custom filter and jail for blocking 404 scanners +- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) — catching PHP webshell/backdoor probes that return 301 on HTTPS-redirecting servers - [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) — access-log-based wp-login.php brute force detection without plugins - [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) — resolving execmem AVC denials from Fail2ban's grep on Fedora - [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) — managing UFW rules, common patterns, troubleshooting @@ -162,6 +163,7 @@ updated: 2026-04-13T10:10 | Date | Article | Domain | |---|---|---| | 2026-04-13 | [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) | Troubleshooting | +| 2026-04-09 | [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) | Self-Hosting | | 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting | | 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | | 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting | diff --git a/SUMMARY.md b/SUMMARY.md index 47a3129..1f74f3c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,6 @@ --- created: 2026-04-02T16:03 -updated: 2026-04-13T10:10 +updated: 2026-04-13T10:16 --- * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) @@ -35,6 +35,7 @@ updated: 2026-04-13T10:10 * [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) * [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) * [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) + * [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) * [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) * [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) * [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) diff --git a/index.md b/index.md index c600623..33cb438 100644 --- a/index.md +++ b/index.md @@ -1,6 +1,6 @@ --- created: 2026-04-06T09:52 -updated: 2026-04-13T10:11 +updated: 2026-04-13T10:16 --- # MajorLinux Tech Wiki — Index @@ -81,6 +81,7 @@ updated: 2026-04-13T10:11 - [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md) — non-root user, SSH key auth, sshd_config, firewall, fail2ban, SpamAssassin - [Standardizing unattended-upgrades with Ansible](02-selfhosting/security/ansible-unattended-upgrades-fleet.md) — fleet-wide automatic security updates across Ubuntu servers - [Fail2ban Custom Jail: Apache 404 Scanner Detection](02-selfhosting/security/fail2ban-apache-404-scanner-jail.md) — custom filter and jail for blocking 404 scanners +- [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) — catching PHP webshell/backdoor probes that return 301 on HTTPS-redirecting servers - [Fail2ban Custom Jail: WordPress Login Brute Force](02-selfhosting/security/fail2ban-wordpress-login-jail.md) — access-log-based wp-login.php brute force detection without plugins - [SELinux: Fixing Fail2ban grep execmem Denial](02-selfhosting/security/selinux-fail2ban-execmem-fix.md) — resolving execmem AVC denials from Fail2ban's grep on Fedora - [UFW Firewall Management](02-selfhosting/security/ufw-firewall-management.md) — managing UFW rules, common patterns, troubleshooting @@ -166,6 +167,7 @@ updated: 2026-04-13T10:11 | Date | Article | Domain | |---|---|---| | 2026-04-13 | [Cron Heartbeat False Alarm: /var/run Cleared by Reboot](05-troubleshooting/cron-heartbeat-tmpfs-reboot-false-alarm.md) | Troubleshooting | +| 2026-04-09 | [Fail2ban Custom Jail: Apache PHP Webshell Probe Detection](02-selfhosting/security/fail2ban-apache-php-probe-jail.md) | Self-Hosting | | 2026-04-08 | [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md) | Troubleshooting | | 2026-04-07 | [SSH Config & Key Management](01-linux/networking/ssh-config-key-management.md) | Linux | | 2026-04-07 | [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md) | Troubleshooting |