MajorWiki/05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md

Fail2ban & UFW Rule Bloat: 30k Rules Slowing Down a VPS

🛑 Problem

A small VPS (1–2 GB RAM) running Fail2ban with permanent bans (bantime = -1) gradually accumulates thousands of UFW DENY rules or nftables entries. Over time this causes:

• High memory usage from Fail2ban (100+ MB RSS)
• Bloated nftables ruleset (30k+ rules) — every incoming packet must traverse the full list
• Netdata alerts flapping on RAM/swap thresholds
• Degraded packet processing performance

---

🔍 Diagnosis

Step 1 — Check Fail2ban memory and thread count

grep -E "VmRSS|VmSwap|Threads" /proc/$(pgrep -ox fail2ban-server)/status

On a small VPS, Fail2ban RSS over 80 MB is a red flag. Thread count scales with jail count (roughly 2 threads per jail + overhead).

---

Step 2 — Count nftables/UFW rules

# Total drop/reject rules in nftables
nft list ruleset | grep -c "reject\|drop"

# UFW rule file size
wc -l /etc/ufw/user.rules

A healthy UFW setup has 10–30 rules. Thousands means manual ufw deny commands or permanent Fail2ban bans have accumulated.

---

Step 3 — Identify dead jails

for jail in $(fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
  total=$(fail2ban-client status $jail | grep "Total banned" | awk '{print $NF}')
  echo "$jail: $total total bans"
done

Jails with zero total bans are dead weight — burning threads and regex cycles for nothing.

---

Step 4 — Check ban policy

grep bantime /etc/fail2ban/jail.local

bantime = -1 means permanent. On a public-facing server, scanner IPs rotate constantly — permanent bans just pile up with no benefit.

---

✅ Solution

Fix 1 — Disable dead jails

Edit /etc/fail2ban/jail.local and set enabled = false for any jail with zero historical bans.

Fix 2 — Switch to time-limited bans

[DEFAULT]
bantime = 30d

[recidive]
bantime = 90d

30 days is long enough to block active campaigns; repeat offenders get 90 days via recidive. Scanner IPs rarely persist beyond a week.

Fix 3 — Flush accumulated bans

fail2ban-client unban --all

Fix 4 — Reset bloated UFW rules

Back up first:

cp /etc/ufw/user.rules /etc/ufw/user.rules.bak
cp /etc/ufw/user6.rules /etc/ufw/user6.rules.bak

Reset and re-add only legitimate ALLOW rules:

ufw --force reset
ufw default deny incoming
ufw default allow outgoing
ufw allow 443/tcp
ufw allow 80/tcp
ufw allow in on tailscale0 to any port 22 comment "SSH via Tailscale"
# Add any other ALLOW rules specific to your server
ufw --force enable

Restart Fail2ban so it re-creates its nftables chains:

systemctl restart fail2ban

---

🔁 Why This Happens

Cause	Effect
bantime = -1 (permanent)	Banned IP list grows forever; nftables rules never expire
Manual ufw deny from <IP>	Each adds a persistent rule to user.rules; survives reboots
Many jails with no hits	Each jail spawns 2+ threads, runs regex against logs continuously
Small VPS (1–2 GB RAM)	Fail2ban + nftables overhead becomes significant fraction of total RAM

---

⚠️ Key Notes

• Deleting UFW rules one-by-one is impractical at scale — ufw delete with 30k rules takes hours. A full reset + re-add is the only efficient path.
• ufw --force reset also resets before.rules and after.rules — UFW auto-backs these up, but verify your custom chains if any exist.
• After flushing bans, expect a brief spike in 4xx responses as scanners that were previously blocked hit Apache again. Fail2ban will re-ban them within minutes.
• The Netdata web_log_1m_successful alert may fire during this window — it will self-clear once bans repopulate.

---

🔎 Quick Diagnostic Commands

# Fail2ban memory usage
grep -E "VmRSS|VmSwap|Threads" /proc/$(pgrep -ox fail2ban-server)/status

# Count nftables rules
nft list ruleset | grep -c "reject\|drop"

# UFW rule count
ufw status numbered | tail -1

# List all jails with ban counts
for jail in $(fail2ban-client status | grep "Jail list" | sed 's/.*://;s/,/ /g'); do
  banned=$(fail2ban-client status $jail | grep "Currently banned" | awk '{print $NF}')
  total=$(fail2ban-client status $jail | grep "Total banned" | awk '{print $NF}')
  echo "$jail: $banned current / $total total"
done

# Flush all bans
fail2ban-client unban --all