diff --git a/02-selfhosting/security/fail2ban-digest-mode-fleet.md b/02-selfhosting/security/fail2ban-digest-mode-fleet.md new file mode 100644 index 0000000..9004784 --- /dev/null +++ b/02-selfhosting/security/fail2ban-digest-mode-fleet.md @@ -0,0 +1,173 @@ +--- +title: "Fail2Ban Digest Mode — Fleet-Wide Quiet Alerts" +domain: selfhosting +category: security +tags: [fail2ban, security, email, ansible, fleet, cron, digest] +status: published +created: 2026-04-22 +updated: 2026-04-22 +--- +# Fail2Ban Digest Mode — Fleet-Wide Quiet Alerts + +## The Problem + +Fail2Ban's default `action_mwl` sends an email for every single ban — IP address, whois lookup, and relevant log lines. On a fleet of 9 servers running 50+ jails, this floods the inbox with hundreds of emails daily. The vast majority are routine scanner bans (probing `.env`, `wp-login.php`, random PHP paths) that require no human attention. + +The signal-to-noise ratio is terrible. Genuinely important events — SSH brute-force attempts, recidive escalations — get buried in the noise. + +## The Solution — Tiered Alert Model + +Three tiers replace the firehose: + +| Tier | Jails | Action | Why | +|------|-------|--------|-----| +| **Immediate email** | `sshd`, `recidive` | `action_mwl` | Security-critical — someone is actively targeting auth or is a repeat offender | +| **Silent ban** | Everything else | `action_` (default) | Ban happens, firewall rule applied, no email sent | +| **Daily digest** | All jails | Cron script at 08:00 UTC | One summary email per host with ban counts across all jails | + +This reduces email volume from hundreds per day to ~10 (one digest per host + occasional sshd/recidive alerts). + +## jail.local Configuration + +### Set silent ban as the default + +In `[DEFAULT]`: + +```ini +[DEFAULT] +action = %(action_)s +``` + +This overrides the stock `action_mwl` for all jails. Bans still happen — the firewall rule is applied — but no email is sent. + +### Keep immediate alerts for critical jails + +```ini +[sshd] +enabled = true +action = %(action_mwl)s + +[recidive] +enabled = true +action = %(action_mwl)s +``` + +### Clean up email subjects with fq-hostname + +By default, fail2ban uses the system FQDN in email subjects. On Tailscale hosts, this produces ugly subjects like `[Fail2Ban] sshd: banned 1.2.3.4 on MajorToot.tail7f2d9.ts.net`. Override it in `[DEFAULT]`: + +```ini +[DEFAULT] +fq-hostname = majortoot +``` + +This sets the `` tag used in action templates, producing cleaner subjects: `[Fail2Ban] sshd: banned 1.2.3.4 on majortoot`. + +## Daily Digest Script + +A shell script at `/usr/local/bin/fail2ban-digest.sh` runs via cron at 08:00 UTC. It does the following: + +1. Queries every active jail via `fail2ban-client status ` +2. Collects four metrics per jail: currently banned, total banned, currently failed, total failed +3. Builds a plain-text email body with one line per jail +4. Separates "active" jails (with bans) from "quiet" jails (zero bans) for quick scanning +5. Sets the subject line to include the total banned count (e.g., `[Fail2Ban Digest] majortoot — 47 total bans`) +6. Sends via `sendmail` + +The script is templated and deployed by Ansible. It lives on each host at `/usr/local/bin/fail2ban-digest.sh`. + +### Cron entry + +``` +0 8 * * * /usr/local/bin/fail2ban-digest.sh +``` + +Managed by `ansible.builtin.cron` — no manual crontab editing needed. + +## Ansible Deployment + +The playbook `configure_fail2ban_digest.yml` deploys the full digest model fleet-wide. + +### What it does + +1. Deploys a Python helper script that performs **section-aware editing** of `jail.local` (see gotchas below) +2. Sets `action = %(action_)s` in `[DEFAULT]` +3. Sets `action = %(action_mwl)s` in `[sshd]` and `[recidive]` +4. Sets `fq-hostname` per host using an override dict +5. Deploys the digest script from a Jinja2 template +6. Creates the cron job via `ansible.builtin.cron` +7. Restarts fail2ban + +### Host-specific overrides + +Two dictionaries in the playbook vars handle per-host customization: + +```yaml +fail2ban_sender_overrides: + majormail: "fail2ban@majorshouse.com" + dcaprod: "fail2ban@dcanalysts.net" + +fail2ban_hostname_overrides: + majortoot: "majortoot" + teelia: "teelia" + majormail: "majormail" +``` + +These feed into the Python editor script and the digest template. + +### Why not lineinfile? + +The playbook uses a Python script for `jail.local` editing instead of Ansible's `lineinfile` module. This is deliberate — see the gotchas section below. + +## Gotchas + +### lineinfile matches stock action definitions + +Using `lineinfile` with a regex like `regexp: '^action\s*=\s*%(action_mwl)s'` is dangerous. In a full `jail.local` that includes stock action *definitions* (copied from `jail.conf`), the regex matches lines like: + +```ini +action_mwl = %(action_mwl)s +``` + +This is the stock definition of the `action_mwl` macro itself — not the `action =` assignment in `[DEFAULT]`. `lineinfile` replaces the wrong line, corrupting the config. Fail2ban then refuses to start because the `action_mwl` macro is undefined. + +**Solution:** Use a Python script that parses INI sections and only modifies the `action` key within the correct `[section]`. This is what the Ansible playbook does. + +### Duplicate action lines crash fail2ban + +If a section ends up with two `action =` lines (e.g., from a botched `lineinfile` run), fail2ban refuses to load: + +``` +option 'action' in section 'DEFAULT' already exists +``` + +The Python editor script handles this by replacing existing keys rather than appending. + +### fq-hostname scope + +Setting `fq-hostname` in `[DEFAULT]` affects all action templates that use the `` tag — including both immediate emails and the digest subject. This is the desired behavior, but be aware that it overrides the system hostname globally within fail2ban. + +## Verifying the Setup + +```bash +# Check that default action is silent +fail2ban-client get DEFAULT action +# Should NOT contain 'sendmail' or 'mail' + +# Check that sshd still sends email +fail2ban-client get sshd action +# Should contain 'sendmail-whois-lines' or similar + +# Trigger a test digest +/usr/local/bin/fail2ban-digest.sh + +# Check cron is installed +crontab -l | grep fail2ban-digest +``` + +## See Also + +- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md) — custom jail for catching 404 scanners +- [fail2ban-wordpress-login-jail](fail2ban-wordpress-login-jail.md) — WordPress brute-force jail +- [ssh-hardening-ansible-fleet](ssh-hardening-ansible-fleet.md) — fleet SSH hardening +- [clamav-fleet-deployment](clamav-fleet-deployment.md) — another fleet-wide security deployment via Ansible diff --git a/SUMMARY.md b/SUMMARY.md index bde282d..f55df4c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,6 @@ --- created: 2026-04-02T16:03 -updated: 2026-04-18T18:48 +updated: 2026-04-21T09:17 --- * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md)