From 961ce75b88aeb72e51c7b925a2b20539c6fd657b Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 17 Apr 2026 21:06:06 -0400 Subject: [PATCH] Add 4 articles: nginx/apache bad-request jails, SSH fleet hardening, Watchtower localhost relay All sourced from 2026-04-17 work sessions: - fail2ban-nginx-bad-request-jail: enable stock jail (just needs wiring) - fail2ban-apache-bad-request-jail: custom filter from scratch, no stock equivalent - ssh-hardening-ansible-fleet: drop-in approach with Fedora/Ubuntu edge cases - watchtower-smtp-localhost-relay: credential-free localhost postfix relay pattern Co-Authored-By: Claude Sonnet 4.6 --- .../docker/watchtower-smtp-localhost-relay.md | 105 +++++++++++++ .../fail2ban-apache-bad-request-jail.md | 127 ++++++++++++++++ .../fail2ban-nginx-bad-request-jail.md | 89 +++++++++++ .../security/ssh-hardening-ansible-fleet.md | 138 ++++++++++++++++++ 4 files changed, 459 insertions(+) create mode 100644 02-selfhosting/docker/watchtower-smtp-localhost-relay.md create mode 100644 02-selfhosting/security/fail2ban-apache-bad-request-jail.md create mode 100644 02-selfhosting/security/fail2ban-nginx-bad-request-jail.md create mode 100644 02-selfhosting/security/ssh-hardening-ansible-fleet.md diff --git a/02-selfhosting/docker/watchtower-smtp-localhost-relay.md b/02-selfhosting/docker/watchtower-smtp-localhost-relay.md new file mode 100644 index 0000000..dbb8416 --- /dev/null +++ b/02-selfhosting/docker/watchtower-smtp-localhost-relay.md @@ -0,0 +1,105 @@ +--- +title: "Watchtower SMTP via Localhost Postfix Relay" +domain: selfhosting +category: docker +tags: [watchtower, docker, smtp, postfix, email, notifications] +status: published +created: 2026-04-17 +updated: 2026-04-17 +--- +# Watchtower SMTP via Localhost Postfix Relay + +## The Problem + +Watchtower supports email notifications via its built-in shoutrrr SMTP driver. The typical setup stores SMTP credentials in the compose file or a separate env file. This creates two failure modes: + +1. **Password rotation breaks notifications silently.** When you rotate your mail server password, Watchtower keeps running but stops sending emails. You only discover it when you notice container updates happened with no notification. +2. **Credentials at rest.** `docker-compose.yml` and `.env` files are often world-readable or checked into git. SMTP passwords stored there are a credential leak waiting to happen. + +The shoutrrr SMTP driver also has a quirk: it attempts AUTH over an unencrypted connection to remote SMTP servers, which most mail servers (correctly) reject with `535 5.7.8 authentication failed` or similar. + +## The Solution + +Route Watchtower's outbound mail through **localhost port 25** using `network_mode: host`. The local Postfix MTA — already running on the host for relay purposes — handles authentication to the upstream mail server. Watchtower never sees a credential. + +``` +Watchtower → localhost:25 (Postfix, trusted via mynetworks — no auth required) + → Postfix → upstream mail server → delivery +``` + +## docker-compose.yml + +```yaml +services: + watchtower: + image: containrrr/watchtower + restart: always + network_mode: host + volumes: + - /var/run/docker.sock:/var/run/docker.sock + environment: + - DOCKER_API_VERSION=1.44 + - WATCHTOWER_CLEANUP=true + - WATCHTOWER_SCHEDULE=0 0 4 * * * + - WATCHTOWER_INCLUDE_STOPPED=false + - WATCHTOWER_NOTIFICATIONS=email + - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com + - WATCHTOWER_NOTIFICATION_EMAIL_TO=you@yourdomain.com + - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=localhost + - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=25 + - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY=true + - WATCHTOWER_NOTIFICATION_EMAIL_DELAY=2 +``` + +**Key settings:** +- `network_mode: host` — required so `localhost` resolves to the host's loopback interface (and port 25). Without this, `localhost` resolves to the container's own loopback, which has no Postfix. +- `EMAIL_SERVER=localhost`, `PORT=25` — target the local Postfix +- `TLS_SKIP_VERIFY=true` — shoutrrr still negotiates STARTTLS even on port 25; a self-signed or expired local Postfix cert is fine to skip +- No `EMAIL_SERVER_USER` or `EMAIL_SERVER_PASSWORD` — Postfix trusts `127.0.0.1` via `mynetworks`, no auth needed + +## Prerequisites + +The host needs a Postfix instance that: +1. Listens on `localhost:25` +2. Includes `127.0.0.0/8` in `mynetworks` so local processes can relay without authentication +3. Is configured to relay outbound to your actual mail server + +This is standard for any host already running a Postfix relay. If Postfix isn't installed, a minimal relay-only config is a few lines in `main.cf`. + +## Why Not Just Use an Env File? + +A separate env file (mode 0600) is better than inline compose, but you still have a credential that breaks on rotation. The localhost relay pattern eliminates the credential entirely. + +| Approach | Credentials stored | Rotation-safe | +|---|---|---| +| Inline in compose | Yes (plaintext, often 0644) | ❌ | +| Separate env file (0600) | Yes (protected but present) | ❌ | +| Localhost Postfix relay | None | ✅ | + +## Testing + +After `docker compose up -d`, check the Watchtower logs for a startup notification: + +```bash +docker logs 2>&1 | head -20 +# Look for: "Sending notification..." +``` + +Confirm Postfix delivered it: + +```bash +grep watchtower /var/log/mail.log | tail -5 +# Look for: status=sent (250 2.0.0 Ok) +``` + +## Gotchas + +- **`network_mode: host` is Linux-only.** Docker Desktop on macOS/Windows doesn't support host networking. This pattern only works on Linux hosts. +- **`network_mode: host` drops port mappings.** Any `ports:` entries are silently ignored under `network_mode: host`. Watchtower doesn't expose ports, so this isn't an issue. +- **Postfix TLS cert warning.** shoutrrr attempts STARTTLS on port 25 regardless. If the local Postfix has a self-signed or expired cert, `TLS_SKIP_VERIFY=true` suppresses the error. For a proper fix, renew the Postfix cert. +- **`WATCHTOWER_DISABLE_CONTAINERS`.** If you run stacks that manage their own updates (Nextcloud AIO, etc.), list those containers here (space-separated) to prevent Watchtower from interfering. + +## See Also + +- [docker-healthchecks](docker-healthchecks.md) +- [debugging-broken-docker-containers](debugging-broken-docker-containers.md) diff --git a/02-selfhosting/security/fail2ban-apache-bad-request-jail.md b/02-selfhosting/security/fail2ban-apache-bad-request-jail.md new file mode 100644 index 0000000..6025c1f --- /dev/null +++ b/02-selfhosting/security/fail2ban-apache-bad-request-jail.md @@ -0,0 +1,127 @@ +--- +title: "Fail2ban Custom Jail: Apache Bad Request Detection" +domain: selfhosting +category: security +tags: [fail2ban, apache, security, firewall, bad-request] +status: published +created: 2026-04-17 +updated: 2026-04-17 +--- +# Fail2ban Custom Jail: Apache Bad Request Detection + +## The Problem + +fail2ban ships a stock `nginx-bad-request` filter for catching malformed HTTP requests (400s), but **there is no Apache equivalent**. Apache servers are left unprotected against the same class of attack: scanners that send garbage request lines to probe for vulnerabilities or overwhelm the access log. + +Unlike the nginx version, this filter has to be written from scratch. + +## The Solution + +Create a custom filter targeting **400 Bad Request** responses in Apache's Combined Log Format, then wire it to a jail. + +### Step 1 — Create the filter + +Create `/etc/fail2ban/filter.d/apache-bad-request.conf`: + +```ini +# Fail2Ban filter: catch 400 Bad Request responses in Apache access logs +# Targets malformed HTTP requests — garbage request lines, empty methods, etc. +# No stock equivalent exists; nginx-bad-request ships with fail2ban but Apache does not. + +[Definition] + +# Match 400 responses in Apache Combined/Common Log Format +failregex = ^ -.*".*" 400 \d+ + +ignoreregex = + +datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z +``` + +### Step 2 — Validate the filter + +Always test before deploying: + +```bash +fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf +``` + +Against a live server under typical traffic this matched **155 lines with zero false positives**. If you see unexpected matches, refine `ignoreregex`. + +### Step 3 — Create the jail drop-in + +Create `/etc/fail2ban/jail.d/apache-bad-request.conf`: + +```ini +[apache-bad-request] +enabled = true +port = http,https +filter = apache-bad-request +logpath = /var/log/apache2/access.log +maxretry = 10 +findtime = 60 +bantime = 1h +``` + +> **Note:** On Fedora/RHEL, the log path may be `/var/log/httpd/access_log`. If your `[DEFAULT]` sets `backend = systemd`, add `backend = polling` to the jail — otherwise it silently ignores `logpath` and reads journald instead. + +### Step 4 — Reload fail2ban + +```bash +systemctl reload fail2ban +fail2ban-client status apache-bad-request +``` + +## Deploy Fleet-Wide with Ansible + +If you run multiple Apache hosts, use Ansible to deploy both the filter and jail atomically: + +```yaml +- name: Deploy apache-bad-request fail2ban filter + ansible.builtin.template: + src: templates/fail2ban_apache_bad_request_filter.conf.j2 + dest: /etc/fail2ban/filter.d/apache-bad-request.conf + notify: Reload fail2ban + +- name: Deploy apache-bad-request fail2ban jail + ansible.builtin.template: + src: templates/fail2ban_apache_bad_request_jail.conf.j2 + dest: /etc/fail2ban/jail.d/apache-bad-request.conf + notify: Reload fail2ban +``` + +## Why Not Use nginx-bad-request on Apache? + +The `nginx-bad-request` filter parses nginx's log format, which differs from Apache's Combined Log Format. The timestamp format, field ordering, and quoting differ enough that the regex won't match. You need a separate filter. + +| | nginx-bad-request | apache-bad-request | +|---|---|---| +| Ships with fail2ban | ✅ Yes | ❌ No — must write custom | +| Log source | nginx access log | Apache access log | +| What it catches | 400 responses (malformed requests) | 400 responses (malformed requests) | +| Regex target | nginx Combined Log Format | Apache Combined Log Format | + +## Diagnostic Commands + +```bash +# Validate filter against live log +fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf + +# Check jail status +fail2ban-client status apache-bad-request + +# Confirm the jail is monitoring the correct log file +fail2ban-client get apache-bad-request logpath + +# Watch bans in real time +tail -f /var/log/fail2ban.log | grep apache-bad-request + +# Count 400s in today's access log +grep '" 400 ' /var/log/apache2/access.log | wc -l +``` + +## See Also + +- [fail2ban-nginx-bad-request-jail](fail2ban-nginx-bad-request-jail.md) — the nginx equivalent (stock filter, just needs wiring) +- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md) — catches 404 probe scanners +- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md) diff --git a/02-selfhosting/security/fail2ban-nginx-bad-request-jail.md b/02-selfhosting/security/fail2ban-nginx-bad-request-jail.md new file mode 100644 index 0000000..9846eb4 --- /dev/null +++ b/02-selfhosting/security/fail2ban-nginx-bad-request-jail.md @@ -0,0 +1,89 @@ +--- +title: "Fail2ban: Enable the nginx-bad-request Jail" +domain: selfhosting +category: security +tags: [fail2ban, nginx, security, firewall, bad-request] +status: published +created: 2026-04-17 +updated: 2026-04-17 +--- +# Fail2ban: Enable the nginx-bad-request Jail + +## The Problem + +Automated scanners sometimes send **malformed HTTP requests** — empty request lines, truncated headers, or garbage data — that nginx rejects with a `400 Bad Request`. These aren't caught by the default fail2ban jails (`nginx-botsearch`, `nginx-http-auth`) because those target URL-probe patterns and auth failures, not raw protocol abuse. + +In a real incident: a single IP (`185.177.72.70`) sent **2,778 malformed requests in ~4 minutes**, driving Netdata's `web_log_1m_bad_requests` to 93.7% and triggering a CRITICAL alert. The neighboring IP (`185.177.72.61`) was already banned — the `/24` was known-bad and operating in shifts. + +## The Solution + +fail2ban ships a `nginx-bad-request` filter out of the box. It's just not wired to a jail by default. Enabling it is a one-step drop-in. + +### Step 1 — Create the jail drop-in + +Create `/etc/fail2ban/jail.d/nginx-bad-request.conf`: + +```ini +[nginx-bad-request] +enabled = true +port = http,https +filter = nginx-bad-request +logpath = /var/log/nginx/access.log +maxretry = 10 +findtime = 60 +bantime = 1h +``` + +**Settings rationale:** +- `maxretry = 10` — a legitimate browser never sends 10 malformed requests; this threshold catches burst scanners immediately +- `findtime = 60` — 60-second window; the attack pattern fires dozens of requests per minute +- `bantime = 1h` — reasonable starting point; pair with `recidive` for repeat offenders + +### Step 2 — Verify the filter matches your log format + +Before reloading, confirm the stock filter matches your nginx logs: + +```bash +fail2ban-regex /var/log/nginx/access.log nginx-bad-request +``` + +In a real-world test against an active server this matched **2,829 lines with zero false positives**. + +### Step 3 — Reload fail2ban + +```bash +systemctl reload fail2ban +fail2ban-client status nginx-bad-request +``` + +You can also ban an IP manually while the jail is loading: + +```bash +fail2ban-client set nginx-bad-request banip 185.177.72.70 +``` + +## Verify It's Working + +```bash +# Check jail status and active bans +fail2ban-client status nginx-bad-request + +# Watch bans in real time +tail -f /var/log/fail2ban.log | grep nginx-bad-request + +# Confirm the jail is monitoring the right file +fail2ban-client get nginx-bad-request logpath +``` + +## Key Notes + +- The stock filter is at `/etc/fail2ban/filter.d/nginx-bad-request.conf` — no need to create it. +- If your `[DEFAULT]` section sets `backend = systemd` (common on Fedora/RHEL), add `backend = polling` to the jail or it will silently ignore `logpath` and monitor journald instead — where nginx doesn't write. +- Make sure your Tailscale subnet (`100.64.0.0/10`) is in `ignoreip` under `[DEFAULT]` to avoid banning your own monitoring. +- This jail targets **400 Bad Request** responses. For 404 scanner detection, see [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md). + +## See Also + +- [fail2ban-apache-bad-request-jail](fail2ban-apache-bad-request-jail.md) — Apache equivalent (no stock filter; custom filter required) +- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md) +- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md) diff --git a/02-selfhosting/security/ssh-hardening-ansible-fleet.md b/02-selfhosting/security/ssh-hardening-ansible-fleet.md new file mode 100644 index 0000000..2f32a71 --- /dev/null +++ b/02-selfhosting/security/ssh-hardening-ansible-fleet.md @@ -0,0 +1,138 @@ +--- +title: "SSH Hardening Fleet-Wide with Ansible" +domain: selfhosting +category: security +tags: [ssh, ansible, security, hardening, fleet] +status: published +created: 2026-04-17 +updated: 2026-04-17 +--- +# SSH Hardening Fleet-Wide with Ansible + +## Overview + +Default SSH daemon settings on both Ubuntu and Fedora/RHEL are permissive. A drop-in configuration file (`/etc/ssh/sshd_config.d/99-hardening.conf`) lets you tighten settings without touching the distro-managed base config — and Ansible can deploy it atomically across every fleet host with a single playbook run. + +## Settings to Change + +| Setting | Default | Hardened | Reason | +|---|---|---|---| +| `PermitRootLogin` | `yes` | `without-password` | Prevent password-based root login; key auth still works for Ansible | +| `X11Forwarding` | `yes` | `no` | Nothing in a typical homelab fleet uses X11 tunneling | +| `AllowTcpForwarding` | `yes` | `no` | Eliminates a tunneling vector if a service account is compromised | +| `MaxAuthTries` | `6` | `3` | Cuts per-connection brute-force attempts in half | +| `LoginGraceTime` | `120` | `30` | Reduces the window for slow-connect attacks | + +## The Drop-in Approach + +Rather than editing `/etc/ssh/sshd_config` directly (which may be managed by the distro or overwritten on upgrades), place overrides in `/etc/ssh/sshd_config.d/99-hardening.conf`. The `Include /etc/ssh/sshd_config.d/*.conf` directive in the base config loads these in alphabetical order, and **first match wins** — so `99-` ensures your overrides come last and take precedence. + +> **Fedora/RHEL gotcha:** Fedora ships `/etc/ssh/sshd_config.d/50-redhat.conf` which sets `X11Forwarding yes`. Because first-match-wins applies, `50-redhat.conf` loads before `99-hardening.conf` and wins. You must patch `50-redhat.conf` in-place before deploying your drop-in, or the X11Forwarding setting will be silently ignored. + +## Ansible Playbook + +```yaml +- name: Harden SSH daemon fleet-wide + hosts: all:!raspbian + become: true + gather_facts: true + + tasks: + + - name: Ensure sshd_config.d directory exists + ansible.builtin.file: + path: /etc/ssh/sshd_config.d + state: directory + owner: root + group: root + mode: '0755' + + - name: Ensure Include directive is present in sshd_config + ansible.builtin.lineinfile: + path: /etc/ssh/sshd_config + line: "Include /etc/ssh/sshd_config.d/*.conf" + insertbefore: BOF + state: present + + # Fedora only: neutralize 50-redhat.conf's X11Forwarding yes + # (first-match-wins means it would override our 99- drop-in) + - name: Comment out X11Forwarding in 50-redhat.conf (Fedora) + ansible.builtin.replace: + path: /etc/ssh/sshd_config.d/50-redhat.conf + regexp: '^(X11Forwarding yes)' + replace: '# \1 # disabled by ansible hardening' + when: ansible_os_family == "RedHat" + ignore_errors: true + + - name: Deploy SSH hardening drop-in + ansible.builtin.copy: + dest: /etc/ssh/sshd_config.d/99-hardening.conf + content: | + # Managed by Ansible — do not edit manually + PermitRootLogin without-password + X11Forwarding no + AllowTcpForwarding no + MaxAuthTries 3 + LoginGraceTime 30 + owner: root + group: root + mode: '0644' + notify: Reload sshd + + - name: Verify effective SSH settings + ansible.builtin.command: + cmd: sshd -T + register: sshd_effective + changed_when: false + + - name: Assert hardened settings are active + ansible.builtin.assert: + that: + - "'permitrootlogin without-password' in sshd_effective.stdout" + - "'x11forwarding no' in sshd_effective.stdout" + - "'allowtcpforwarding no' in sshd_effective.stdout" + - "'maxauthtries 3' in sshd_effective.stdout" + - "'logingracetime 30' in sshd_effective.stdout" + fail_msg: "One or more SSH hardening settings not effective — check for conflicting config" + when: not ansible_check_mode + + handlers: + + - name: Reload sshd + ansible.builtin.service: + # Ubuntu/Debian: 'ssh' | Fedora/RHEL: 'sshd' + name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}" + state: reloaded +``` + +## Edge Cases + +**Ubuntu vs Fedora service name:** The SSH daemon is `ssh` on Debian/Ubuntu and `sshd` on Fedora/RHEL. The handler uses `ansible_os_family` to pick the right name automatically. + +**Missing Include directive:** Some minimal installs don't have `Include /etc/ssh/sshd_config.d/*.conf` in their base config. The `lineinfile` task adds it if absent. Without this, the drop-in directory exists but is never loaded. + +**Fedora's 50-redhat.conf:** Sets `X11Forwarding yes` with first-match priority. The playbook patches it before deploying the drop-in. + +**`sshd -T` in check mode:** `sshd -T` reads the *current* running config, not the pending changes. The assert task is guarded with `when: not ansible_check_mode` to prevent false failures during dry runs. + +**PermitRootLogin on hosts that already had it set:** Some hosts (e.g., those managed by another tool) may already have `PermitRootLogin without-password` set elsewhere. The drop-in still applies cleanly — it just becomes a no-op for that setting. + +## Verify Manually + +```bash +# Check effective settings on any host +ssh root@ "sshd -T | grep -E 'permitrootlogin|x11forwarding|allowtcpforwarding|maxauthtries|logingracetime'" + +# Expected: +# permitrootlogin without-password +# x11forwarding no +# allowtcpforwarding no +# maxauthtries 3 +# logingracetime 30 +``` + +## See Also + +- [linux-server-hardening-checklist](linux-server-hardening-checklist.md) +- [ansible-unattended-upgrades-fleet](ansible-unattended-upgrades-fleet.md) +- [ufw-firewall-management](ufw-firewall-management.md)