majorlinux/MajorWiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add 4 articles: nginx/apache bad-request jails, SSH fleet hardening, Watchtower localhost relay

Browse Source 
All sourced from 2026-04-17 work sessions:
- fail2ban-nginx-bad-request-jail: enable stock jail (just needs wiring)
- fail2ban-apache-bad-request-jail: custom filter from scratch, no stock equivalent
- ssh-hardening-ansible-fleet: drop-in approach with Fedora/Ubuntu edge cases
- watchtower-smtp-localhost-relay: credential-free localhost postfix relay pattern

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
MajorLinux
2026-04-17 21:06:06 -04:00
parent 9c1a8c95d5
commit 961ce75b88
4 changed files with 459 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
02-selfhosting/docker/watchtower-smtp-localhost-relay.md Normal file
View File

@@ -0,0 +1,105 @@
---
title: "Watchtower SMTP via Localhost Postfix Relay"
domain: selfhosting
category: docker
tags: [watchtower, docker, smtp, postfix, email, notifications]
status: published
created: 2026-04-17
updated: 2026-04-17
---
# Watchtower SMTP via Localhost Postfix Relay
## The Problem
Watchtower supports email notifications via its built-in shoutrrr SMTP driver. The typical setup stores SMTP credentials in the compose file or a separate env file. This creates two failure modes:
1. **Password rotation breaks notifications silently.** When you rotate your mail server password, Watchtower keeps running but stops sending emails. You only discover it when you notice container updates happened with no notification.
2. **Credentials at rest.** `docker-compose.yml` and `.env` files are often world-readable or checked into git. SMTP passwords stored there are a credential leak waiting to happen.
The shoutrrr SMTP driver also has a quirk: it attempts AUTH over an unencrypted connection to remote SMTP servers, which most mail servers (correctly) reject with `535 5.7.8 authentication failed` or similar.
## The Solution
Route Watchtower's outbound mail through **localhost port 25** using `network_mode: host`. The local Postfix MTA — already running on the host for relay purposes — handles authentication to the upstream mail server. Watchtower never sees a credential.
```
Watchtower → localhost:25 (Postfix, trusted via mynetworks — no auth required)
→ Postfix → upstream mail server → delivery
```
## docker-compose.yml
```yaml
services:
watchtower:
image: containrrr/watchtower
restart: always
network_mode: host
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- DOCKER_API_VERSION=1.44
- WATCHTOWER_CLEANUP=true
- WATCHTOWER_SCHEDULE=0 0 4 * * *
- WATCHTOWER_INCLUDE_STOPPED=false
- WATCHTOWER_NOTIFICATIONS=email
- WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
- WATCHTOWER_NOTIFICATION_EMAIL_TO=you@yourdomain.com
- WATCHTOWER_NOTIFICATION_EMAIL_SERVER=localhost
- WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=25
- WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY=true
- WATCHTOWER_NOTIFICATION_EMAIL_DELAY=2
```
**Key settings:**
- `network_mode: host` — required so `localhost` resolves to the host's loopback interface (and port 25). Without this, `localhost` resolves to the container's own loopback, which has no Postfix.
- `EMAIL_SERVER=localhost`, `PORT=25` — target the local Postfix
- `TLS_SKIP_VERIFY=true` — shoutrrr still negotiates STARTTLS even on port 25; a self-signed or expired local Postfix cert is fine to skip
- No `EMAIL_SERVER_USER` or `EMAIL_SERVER_PASSWORD` — Postfix trusts `127.0.0.1` via `mynetworks`, no auth needed
## Prerequisites
The host needs a Postfix instance that:
1. Listens on `localhost:25`
2. Includes `127.0.0.0/8` in `mynetworks` so local processes can relay without authentication
3. Is configured to relay outbound to your actual mail server
This is standard for any host already running a Postfix relay. If Postfix isn't installed, a minimal relay-only config is a few lines in `main.cf`.
## Why Not Just Use an Env File?
A separate env file (mode 0600) is better than inline compose, but you still have a credential that breaks on rotation. The localhost relay pattern eliminates the credential entirely.
| Approach | Credentials stored | Rotation-safe |
|---|---|---|
| Inline in compose | Yes (plaintext, often 0644) | ❌ |
| Separate env file (0600) | Yes (protected but present) | ❌ |
| Localhost Postfix relay | None | ✅ |
## Testing
After `docker compose up -d`, check the Watchtower logs for a startup notification:
```bash
docker logs <watchtower-container-name> 2>&1 | head -20
# Look for: "Sending notification..."
```
Confirm Postfix delivered it:
```bash
grep watchtower /var/log/mail.log | tail -5
# Look for: status=sent (250 2.0.0 Ok)
```
## Gotchas
- **`network_mode: host` is Linux-only.** Docker Desktop on macOS/Windows doesn't support host networking. This pattern only works on Linux hosts.
- **`network_mode: host` drops port mappings.** Any `ports:` entries are silently ignored under `network_mode: host`. Watchtower doesn't expose ports, so this isn't an issue.
- **Postfix TLS cert warning.** shoutrrr attempts STARTTLS on port 25 regardless. If the local Postfix has a self-signed or expired cert, `TLS_SKIP_VERIFY=true` suppresses the error. For a proper fix, renew the Postfix cert.
- **`WATCHTOWER_DISABLE_CONTAINERS`.** If you run stacks that manage their own updates (Nextcloud AIO, etc.), list those containers here (space-separated) to prevent Watchtower from interfering.
## See Also
- [docker-healthchecks](docker-healthchecks.md)
- [debugging-broken-docker-containers](debugging-broken-docker-containers.md)

127
02-selfhosting/security/fail2ban-apache-bad-request-jail.md Normal file
View File

@@ -0,0 +1,127 @@
---
title: "Fail2ban Custom Jail: Apache Bad Request Detection"
domain: selfhosting
category: security
tags: [fail2ban, apache, security, firewall, bad-request]
status: published
created: 2026-04-17
updated: 2026-04-17
---
# Fail2ban Custom Jail: Apache Bad Request Detection
## The Problem
fail2ban ships a stock `nginx-bad-request` filter for catching malformed HTTP requests (400s), but **there is no Apache equivalent**. Apache servers are left unprotected against the same class of attack: scanners that send garbage request lines to probe for vulnerabilities or overwhelm the access log.
Unlike the nginx version, this filter has to be written from scratch.
## The Solution
Create a custom filter targeting **400 Bad Request** responses in Apache's Combined Log Format, then wire it to a jail.
### Step 1 — Create the filter
Create `/etc/fail2ban/filter.d/apache-bad-request.conf`:
```ini
# Fail2Ban filter: catch 400 Bad Request responses in Apache access logs
# Targets malformed HTTP requests — garbage request lines, empty methods, etc.
# No stock equivalent exists; nginx-bad-request ships with fail2ban but Apache does not.
[Definition]
# Match 400 responses in Apache Combined/Common Log Format
failregex = ^<HOST> -.*".*" 400 \d+
ignoreregex =
datepattern = %%d/%%b/%%Y:%%H:%%M:%%S %%z
```
### Step 2 — Validate the filter
Always test before deploying:
```bash
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf
```
Against a live server under typical traffic this matched **155 lines with zero false positives**. If you see unexpected matches, refine `ignoreregex`.
### Step 3 — Create the jail drop-in
Create `/etc/fail2ban/jail.d/apache-bad-request.conf`:
```ini
[apache-bad-request]
enabled = true
port = http,https
filter = apache-bad-request
logpath = /var/log/apache2/access.log
maxretry = 10
findtime = 60
bantime = 1h
```
> **Note:** On Fedora/RHEL, the log path may be `/var/log/httpd/access_log`. If your `[DEFAULT]` sets `backend = systemd`, add `backend = polling` to the jail — otherwise it silently ignores `logpath` and reads journald instead.
### Step 4 — Reload fail2ban
```bash
systemctl reload fail2ban
fail2ban-client status apache-bad-request
```
## Deploy Fleet-Wide with Ansible
If you run multiple Apache hosts, use Ansible to deploy both the filter and jail atomically:
```yaml
- name: Deploy apache-bad-request fail2ban filter
ansible.builtin.template:
src: templates/fail2ban_apache_bad_request_filter.conf.j2
dest: /etc/fail2ban/filter.d/apache-bad-request.conf
notify: Reload fail2ban
- name: Deploy apache-bad-request fail2ban jail
ansible.builtin.template:
src: templates/fail2ban_apache_bad_request_jail.conf.j2
dest: /etc/fail2ban/jail.d/apache-bad-request.conf
notify: Reload fail2ban
```
## Why Not Use nginx-bad-request on Apache?
The `nginx-bad-request` filter parses nginx's log format, which differs from Apache's Combined Log Format. The timestamp format, field ordering, and quoting differ enough that the regex won't match. You need a separate filter.
| | nginx-bad-request | apache-bad-request |
|---|---|---|
| Ships with fail2ban | ✅ Yes | ❌ No — must write custom |
| Log source | nginx access log | Apache access log |
| What it catches | 400 responses (malformed requests) | 400 responses (malformed requests) |
| Regex target | nginx Combined Log Format | Apache Combined Log Format |
## Diagnostic Commands
```bash
# Validate filter against live log
fail2ban-regex /var/log/apache2/access.log /etc/fail2ban/filter.d/apache-bad-request.conf
# Check jail status
fail2ban-client status apache-bad-request
# Confirm the jail is monitoring the correct log file
fail2ban-client get apache-bad-request logpath
# Watch bans in real time
tail -f /var/log/fail2ban.log | grep apache-bad-request
# Count 400s in today's access log
grep '" 400 ' /var/log/apache2/access.log | wc -l
```
## See Also
- [fail2ban-nginx-bad-request-jail](fail2ban-nginx-bad-request-jail.md) — the nginx equivalent (stock filter, just needs wiring)
- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md) — catches 404 probe scanners
- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md)

89
02-selfhosting/security/fail2ban-nginx-bad-request-jail.md Normal file
View File

@@ -0,0 +1,89 @@
---
title: "Fail2ban: Enable the nginx-bad-request Jail"
domain: selfhosting
category: security
tags: [fail2ban, nginx, security, firewall, bad-request]
status: published
created: 2026-04-17
updated: 2026-04-17
---
# Fail2ban: Enable the nginx-bad-request Jail
## The Problem
Automated scanners sometimes send **malformed HTTP requests** — empty request lines, truncated headers, or garbage data — that nginx rejects with a `400 Bad Request`. These aren't caught by the default fail2ban jails (`nginx-botsearch`, `nginx-http-auth`) because those target URL-probe patterns and auth failures, not raw protocol abuse.
In a real incident: a single IP (`185.177.72.70`) sent **2,778 malformed requests in ~4 minutes**, driving Netdata's `web_log_1m_bad_requests` to 93.7% and triggering a CRITICAL alert. The neighboring IP (`185.177.72.61`) was already banned — the `/24` was known-bad and operating in shifts.
## The Solution
fail2ban ships a `nginx-bad-request` filter out of the box. It's just not wired to a jail by default. Enabling it is a one-step drop-in.
### Step 1 — Create the jail drop-in
Create `/etc/fail2ban/jail.d/nginx-bad-request.conf`:
```ini
[nginx-bad-request]
enabled = true
port = http,https
filter = nginx-bad-request
logpath = /var/log/nginx/access.log
maxretry = 10
findtime = 60
bantime = 1h
```
**Settings rationale:**
- `maxretry = 10` — a legitimate browser never sends 10 malformed requests; this threshold catches burst scanners immediately
- `findtime = 60` — 60-second window; the attack pattern fires dozens of requests per minute
- `bantime = 1h` — reasonable starting point; pair with `recidive` for repeat offenders
### Step 2 — Verify the filter matches your log format
Before reloading, confirm the stock filter matches your nginx logs:
```bash
fail2ban-regex /var/log/nginx/access.log nginx-bad-request
```
In a real-world test against an active server this matched **2,829 lines with zero false positives**.
### Step 3 — Reload fail2ban
```bash
systemctl reload fail2ban
fail2ban-client status nginx-bad-request
```
You can also ban an IP manually while the jail is loading:
```bash
fail2ban-client set nginx-bad-request banip 185.177.72.70
```
## Verify It's Working
```bash
# Check jail status and active bans
fail2ban-client status nginx-bad-request
# Watch bans in real time
tail -f /var/log/fail2ban.log | grep nginx-bad-request
# Confirm the jail is monitoring the right file
fail2ban-client get nginx-bad-request logpath
```
## Key Notes
- The stock filter is at `/etc/fail2ban/filter.d/nginx-bad-request.conf` — no need to create it.
- If your `[DEFAULT]` section sets `backend = systemd` (common on Fedora/RHEL), add `backend = polling` to the jail or it will silently ignore `logpath` and monitor journald instead — where nginx doesn't write.
- Make sure your Tailscale subnet (`100.64.0.0/10`) is in `ignoreip` under `[DEFAULT]` to avoid banning your own monitoring.
- This jail targets **400 Bad Request** responses. For 404 scanner detection, see [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md).
## See Also
- [fail2ban-apache-bad-request-jail](fail2ban-apache-bad-request-jail.md) — Apache equivalent (no stock filter; custom filter required)
- [fail2ban-apache-404-scanner-jail](fail2ban-apache-404-scanner-jail.md)
- [fail2ban-apache-php-probe-jail](fail2ban-apache-php-probe-jail.md)

138
02-selfhosting/security/ssh-hardening-ansible-fleet.md Normal file
View File

@@ -0,0 +1,138 @@
---
title: "SSH Hardening Fleet-Wide with Ansible"
domain: selfhosting
category: security
tags: [ssh, ansible, security, hardening, fleet]
status: published
created: 2026-04-17
updated: 2026-04-17
---
# SSH Hardening Fleet-Wide with Ansible
## Overview
Default SSH daemon settings on both Ubuntu and Fedora/RHEL are permissive. A drop-in configuration file (`/etc/ssh/sshd_config.d/99-hardening.conf`) lets you tighten settings without touching the distro-managed base config — and Ansible can deploy it atomically across every fleet host with a single playbook run.
## Settings to Change
| Setting | Default | Hardened | Reason |
|---|---|---|---|
| `PermitRootLogin` | `yes` | `without-password` | Prevent password-based root login; key auth still works for Ansible |
| `X11Forwarding` | `yes` | `no` | Nothing in a typical homelab fleet uses X11 tunneling |
| `AllowTcpForwarding` | `yes` | `no` | Eliminates a tunneling vector if a service account is compromised |
| `MaxAuthTries` | `6` | `3` | Cuts per-connection brute-force attempts in half |
| `LoginGraceTime` | `120` | `30` | Reduces the window for slow-connect attacks |
## The Drop-in Approach
Rather than editing `/etc/ssh/sshd_config` directly (which may be managed by the distro or overwritten on upgrades), place overrides in `/etc/ssh/sshd_config.d/99-hardening.conf`. The `Include /etc/ssh/sshd_config.d/*.conf` directive in the base config loads these in alphabetical order, and **first match wins** — so `99-` ensures your overrides come last and take precedence.
> **Fedora/RHEL gotcha:** Fedora ships `/etc/ssh/sshd_config.d/50-redhat.conf` which sets `X11Forwarding yes`. Because first-match-wins applies, `50-redhat.conf` loads before `99-hardening.conf` and wins. You must patch `50-redhat.conf` in-place before deploying your drop-in, or the X11Forwarding setting will be silently ignored.
## Ansible Playbook
```yaml
- name: Harden SSH daemon fleet-wide
hosts: all:!raspbian
become: true
gather_facts: true
tasks:
- name: Ensure sshd_config.d directory exists
ansible.builtin.file:
path: /etc/ssh/sshd_config.d
state: directory
owner: root
group: root
mode: '0755'
- name: Ensure Include directive is present in sshd_config
ansible.builtin.lineinfile:
path: /etc/ssh/sshd_config
line: "Include /etc/ssh/sshd_config.d/*.conf"
insertbefore: BOF
state: present
# Fedora only: neutralize 50-redhat.conf's X11Forwarding yes
# (first-match-wins means it would override our 99- drop-in)
- name: Comment out X11Forwarding in 50-redhat.conf (Fedora)
ansible.builtin.replace:
path: /etc/ssh/sshd_config.d/50-redhat.conf
regexp: '^(X11Forwarding yes)'
replace: '# \1 # disabled by ansible hardening'
when: ansible_os_family == "RedHat"
ignore_errors: true
- name: Deploy SSH hardening drop-in
ansible.builtin.copy:
dest: /etc/ssh/sshd_config.d/99-hardening.conf
content: |
# Managed by Ansible — do not edit manually
PermitRootLogin without-password
X11Forwarding no
AllowTcpForwarding no
MaxAuthTries 3
LoginGraceTime 30
owner: root
group: root
mode: '0644'
notify: Reload sshd
- name: Verify effective SSH settings
ansible.builtin.command:
cmd: sshd -T
register: sshd_effective
changed_when: false
- name: Assert hardened settings are active
ansible.builtin.assert:
that:
- "'permitrootlogin without-password' in sshd_effective.stdout"
- "'x11forwarding no' in sshd_effective.stdout"
- "'allowtcpforwarding no' in sshd_effective.stdout"
- "'maxauthtries 3' in sshd_effective.stdout"
- "'logingracetime 30' in sshd_effective.stdout"
fail_msg: "One or more SSH hardening settings not effective — check for conflicting config"
when: not ansible_check_mode
handlers:
- name: Reload sshd
ansible.builtin.service:
# Ubuntu/Debian: 'ssh' | Fedora/RHEL: 'sshd'
name: "{{ 'ssh' if ansible_os_family == 'Debian' else 'sshd' }}"
state: reloaded
```
## Edge Cases
**Ubuntu vs Fedora service name:** The SSH daemon is `ssh` on Debian/Ubuntu and `sshd` on Fedora/RHEL. The handler uses `ansible_os_family` to pick the right name automatically.
**Missing Include directive:** Some minimal installs don't have `Include /etc/ssh/sshd_config.d/*.conf` in their base config. The `lineinfile` task adds it if absent. Without this, the drop-in directory exists but is never loaded.
**Fedora's 50-redhat.conf:** Sets `X11Forwarding yes` with first-match priority. The playbook patches it before deploying the drop-in.
**`sshd -T` in check mode:** `sshd -T` reads the *current* running config, not the pending changes. The assert task is guarded with `when: not ansible_check_mode` to prevent false failures during dry runs.
**PermitRootLogin on hosts that already had it set:** Some hosts (e.g., those managed by another tool) may already have `PermitRootLogin without-password` set elsewhere. The drop-in still applies cleanly — it just becomes a no-op for that setting.
## Verify Manually
```bash
# Check effective settings on any host
ssh root@<host> "sshd -T | grep -E 'permitrootlogin|x11forwarding|allowtcpforwarding|maxauthtries|logingracetime'"
# Expected:
# permitrootlogin without-password
# x11forwarding no
# allowtcpforwarding no
# maxauthtries 3
# logingracetime 30
```
## See Also
- [linux-server-hardening-checklist](linux-server-hardening-checklist.md)
- [ansible-unattended-upgrades-fleet](ansible-unattended-upgrades-fleet.md)
- [ufw-firewall-management](ufw-firewall-management.md)