Documents the 2026-04-09 scanner incident where 301-redirected PHP probes bypassed the existing apache-404scan jail, leaving the scanner unbanned and firing Netdata web_log_1m_redirects alerts. New jail catches 301/302/ 403/404 PHP responses while excluding legitimate WordPress endpoints. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
147 lines
5.5 KiB
Markdown
147 lines
5.5 KiB
Markdown
---
|
||
title: "Fail2ban Custom Jail: Apache PHP Webshell Probe Detection"
|
||
domain: selfhosting
|
||
category: security
|
||
tags:
|
||
- fail2ban
|
||
- apache
|
||
- security
|
||
- php
|
||
- webshell
|
||
- scanner
|
||
status: published
|
||
created: 2026-04-09
|
||
updated: 2026-04-13T10:15
|
||
---
|
||
# Fail2ban Custom Jail: Apache PHP Webshell Probe Detection
|
||
|
||
## The Problem
|
||
|
||
Automated scanners flood web servers with rapid-fire requests for non-existent `.php` files — `bless.php`, `alfa.php`, `lock360.php`, `about.php`, `cgi-bin/bypass.php`, and hundreds of others. These are classic **webshell/backdoor probes** looking for compromised PHP files left behind by prior attackers.
|
||
|
||
On servers that force HTTPS (or have HTTP→HTTPS redirects in place), these probes often return **301 Moved Permanently** instead of 404. That causes three problems:
|
||
|
||
1. **The `apache-404scan` jail misses them** — it only matches 404 responses
|
||
2. **Netdata fires false `web_log_1m_redirects` alerts** — the redirect ratio spikes to 96%+ during scans
|
||
3. **The scanner is never banned**, and will return repeatedly
|
||
|
||
This was the exact trigger for the 2026-04-09 `[MajorLinux] Web Log Alert` incident where `45.86.202.224` sent 202 PHP probe requests in a few minutes, all returning 301.
|
||
|
||
## The Solution
|
||
|
||
Create a custom Fail2ban filter that matches **any `.php` request returning a redirect, forbidden, or not-found response** — while excluding legitimate WordPress PHP endpoints.
|
||
|
||
### Step 1 — Create the filter
|
||
|
||
Create `/etc/fail2ban/filter.d/apache-php-probe.conf`:
|
||
|
||
```ini
|
||
# Fail2Ban filter to catch PHP file probing (webshell/backdoor scanners)
|
||
# These requests hit non-existent .php files and get 301/302/403/404 responses
|
||
|
||
[Definition]
|
||
|
||
failregex = ^<HOST> -.*"(GET|POST|HEAD) /[^ ]*\.php[^ ]* HTTP/[0-9.]+" (301|302|403|404) \d+
|
||
|
||
ignoreregex = ^<HOST> -.*(wp-cron\.php|xmlrpc\.php|wp-login\.php|wp-admin|index\.php|wp-comments-post\.php)
|
||
|
||
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
|
||
```
|
||
|
||
**Why the ignoreregex matters:** Legitimate WordPress traffic hits `wp-cron.php`, `xmlrpc.php` (often 403-blocked on hardened sites), `wp-login.php`, and `index.php` constantly. Without exclusions the jail would ban your own WordPress admins. Note that `wp-login.php` brute force is caught separately by the `wordpress` jail.
|
||
|
||
### Step 2 — Add the jail
|
||
|
||
Add to `/etc/fail2ban/jail.local`:
|
||
|
||
```ini
|
||
[apache-php-probe]
|
||
enabled = true
|
||
port = http,https
|
||
filter = apache-php-probe
|
||
logpath = /var/log/apache2/access.log
|
||
maxretry = 5
|
||
findtime = 1m
|
||
bantime = 48h
|
||
```
|
||
|
||
**5 hits in 1 minute** is tight — scanners fire 20–200 PHP probes in seconds, while a real user hitting one broken PHP link won't trip the threshold. The 48-hour bantime is longer than `apache-404scan`'s 24h because PHP webshell scanning is a stronger signal of malicious intent.
|
||
|
||
### Step 3 — Test the regex
|
||
|
||
```bash
|
||
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-php-probe.conf
|
||
```
|
||
|
||
Verify it matches the scanner requests and does **not** match legitimate WordPress traffic.
|
||
|
||
### Step 4 — Reload Fail2ban
|
||
|
||
```bash
|
||
systemctl restart fail2ban
|
||
fail2ban-client status apache-php-probe
|
||
```
|
||
|
||
## Why This Complements `apache-404scan`
|
||
|
||
| Jail | Catches | Misses |
|
||
|---|---|---|
|
||
| `apache-404scan` | Any 404 (config file probes, `.env`, random paths) | PHP probes redirected to HTTPS (301) |
|
||
| **`apache-php-probe`** | **PHP webshell probes (301/302/403/404)** | Non-`.php` probes |
|
||
|
||
Running both jails together covers:
|
||
- **HTTP→HTTPS redirected PHP probes** (301 responses)
|
||
- **Directly-served PHP probes** (404 responses)
|
||
- **Blocked PHP paths** like `xmlrpc.php` in non-WP contexts (403 responses)
|
||
|
||
## Pair With Recidive
|
||
|
||
The `recidive` jail catches repeat offenders across all jails:
|
||
|
||
```ini
|
||
[recidive]
|
||
enabled = true
|
||
bantime = -1
|
||
findtime = 86400
|
||
maxretry = 3
|
||
```
|
||
|
||
A scanner that trips `apache-php-probe` three times in 24 hours gets a **permanent** firewall-level ban.
|
||
|
||
## Manual IP Blocking via UFW
|
||
|
||
For known scanners you want to block immediately without waiting for the jail to trip, use UFW:
|
||
|
||
```bash
|
||
# Insert at top of rule list (priority over Apache ALLOW rules)
|
||
ufw insert 1 deny from <IP> to any comment "PHP webshell scanner YYYY-MM-DD"
|
||
```
|
||
|
||
This bypasses fail2ban entirely and is useful for:
|
||
- Scanners you spot in logs after the fact
|
||
- Known-malicious subnets from threat intel
|
||
- Entire CIDR blocks (`ufw insert 1 deny from 45.86.202.0/24`)
|
||
|
||
## Quick Diagnostic Commands
|
||
|
||
```bash
|
||
# Count recent PHP probes returning 301/403/404
|
||
awk '/09\/Apr\/2026:18:/ && /\.php/ && ($9==301 || $9==403 || $9==404)' /var/log/apache2/access.log | wc -l
|
||
|
||
# Top probed PHP filenames (useful for writing additional ignoreregex)
|
||
grep '\.php' /var/log/apache2/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20
|
||
|
||
# Top scanner IPs by PHP probe count
|
||
grep '\.php' /var/log/apache2/access.log | awk '$9 ~ /^(301|403|404)$/ {print $1}' | sort | uniq -c | sort -rn | head -10
|
||
|
||
# Watch bans in real time
|
||
tail -f /var/log/fail2ban.log | grep apache-php-probe
|
||
```
|
||
|
||
## Key Notes
|
||
|
||
- **This jail only makes sense on servers that redirect HTTP→HTTPS.** On plain-HTTPS-only servers, PHP probes return 404 and `apache-404scan` already catches them.
|
||
- **Add your own WordPress plugin paths to `ignoreregex`** if you use non-standard endpoints (e.g., custom admin URLs, REST API `.php` handlers).
|
||
- **This filter pairs naturally with Netdata `web_log_1m_redirects` alerts** — during a scan, Netdata fires first (threshold crossed), then fail2ban bans the IP within seconds.
|
||
- Also see: [Fail2ban Custom Jail: Apache 404 Scanner Detection](fail2ban-apache-404-scanner-jail.md) for the sibling 404-based filter.
|