Add 4 articles: nginx/apache bad-request jails, SSH fleet hardening, Watchtower localhost relay

All sourced from 2026-04-17 work sessions: - fail2ban-nginx-bad-request-jail: enable stock jail (just needs wiring) - fail2ban-apache-bad-request-jail: custom filter from scratch, no stock equivalent - ssh-hardening-ansible-fleet: drop-in approach with Fedora/Ubuntu edge cases - watchtower-smtp-localhost-relay: credential-free localhost postfix relay pattern Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-17 21:06:06 -04:00
parent 9c1a8c95d5
commit 961ce75b88
4 changed files with 459 additions and 0 deletions
--- a/02-selfhosting/docker/watchtower-smtp-localhost-relay.md
+++ b/02-selfhosting/docker/watchtower-smtp-localhost-relay.md
@@ -0,0 +1,105 @@
+---
+title: "Watchtower SMTP via Localhost Postfix Relay"
+domain: selfhosting
+category: docker
+tags: [watchtower, docker, smtp, postfix, email, notifications]
+status: published
+created: 2026-04-17
+updated: 2026-04-17
+---
+# Watchtower SMTP via Localhost Postfix Relay
+
+## The Problem
+
+Watchtower supports email notifications via its built-in shoutrrr SMTP driver. The typical setup stores SMTP credentials in the compose file or a separate env file. This creates two failure modes:
+
+1. **Password rotation breaks notifications silently.** When you rotate your mail server password, Watchtower keeps running but stops sending emails. You only discover it when you notice container updates happened with no notification.
+2. **Credentials at rest.** `docker-compose.yml` and `.env` files are often world-readable or checked into git. SMTP passwords stored there are a credential leak waiting to happen.
+
+The shoutrrr SMTP driver also has a quirk: it attempts AUTH over an unencrypted connection to remote SMTP servers, which most mail servers (correctly) reject with `535 5.7.8 authentication failed` or similar.
+
+## The Solution
+
+Route Watchtower's outbound mail through **localhost port 25** using `network_mode: host`. The local Postfix MTA — already running on the host for relay purposes — handles authentication to the upstream mail server. Watchtower never sees a credential.
+
+```
+Watchtower → localhost:25 (Postfix, trusted via mynetworks — no auth required)
+  → Postfix → upstream mail server → delivery
+```
+
+## docker-compose.yml
+
+```yaml
+services:
+  watchtower:
+    image: containrrr/watchtower
+    restart: always
+    network_mode: host
+    volumes:
+      - /var/run/docker.sock:/var/run/docker.sock
+    environment:
+      - DOCKER_API_VERSION=1.44
+      - WATCHTOWER_CLEANUP=true
+      - WATCHTOWER_SCHEDULE=0 0 4 * * *
+      - WATCHTOWER_INCLUDE_STOPPED=false
+      - WATCHTOWER_NOTIFICATIONS=email
+      - WATCHTOWER_NOTIFICATION_EMAIL_FROM=watchtower@yourdomain.com
+      - WATCHTOWER_NOTIFICATION_EMAIL_TO=you@yourdomain.com
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=localhost
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=25
+      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_TLS_SKIP_VERIFY=true
+      - WATCHTOWER_NOTIFICATION_EMAIL_DELAY=2
+```
+
+**Key settings:**
+- `network_mode: host` — required so `localhost` resolves to the host's loopback interface (and port 25). Without this, `localhost` resolves to the container's own loopback, which has no Postfix.
+- `EMAIL_SERVER=localhost`, `PORT=25` — target the local Postfix
+- `TLS_SKIP_VERIFY=true` — shoutrrr still negotiates STARTTLS even on port 25; a self-signed or expired local Postfix cert is fine to skip
+- No `EMAIL_SERVER_USER` or `EMAIL_SERVER_PASSWORD` — Postfix trusts `127.0.0.1` via `mynetworks`, no auth needed
+
+## Prerequisites
+
+The host needs a Postfix instance that:
+1. Listens on `localhost:25`
+2. Includes `127.0.0.0/8` in `mynetworks` so local processes can relay without authentication
+3. Is configured to relay outbound to your actual mail server
+
+This is standard for any host already running a Postfix relay. If Postfix isn't installed, a minimal relay-only config is a few lines in `main.cf`.
+
+## Why Not Just Use an Env File?
+
+A separate env file (mode 0600) is better than inline compose, but you still have a credential that breaks on rotation. The localhost relay pattern eliminates the credential entirely.
+
+| Approach | Credentials stored | Rotation-safe |
+|---|---|---|
+| Inline in compose | Yes (plaintext, often 0644) | ❌ |
+| Separate env file (0600) | Yes (protected but present) | ❌ |
+| Localhost Postfix relay | None | ✅ |
+
+## Testing
+
+After `docker compose up -d`, check the Watchtower logs for a startup notification:
+
+```bash
+docker logs <watchtower-container-name> 2>&1 | head -20
+# Look for: "Sending notification..."
+```
+
+Confirm Postfix delivered it:
+
+```bash
+grep watchtower /var/log/mail.log | tail -5
+# Look for: status=sent (250 2.0.0 Ok)
+```
+
+## Gotchas
+
+- **`network_mode: host` is Linux-only.** Docker Desktop on macOS/Windows doesn't support host networking. This pattern only works on Linux hosts.
+- **`network_mode: host` drops port mappings.** Any `ports:` entries are silently ignored under `network_mode: host`. Watchtower doesn't expose ports, so this isn't an issue.
+- **Postfix TLS cert warning.** shoutrrr attempts STARTTLS on port 25 regardless. If the local Postfix has a self-signed or expired cert, `TLS_SKIP_VERIFY=true` suppresses the error. For a proper fix, renew the Postfix cert.
+- **`WATCHTOWER_DISABLE_CONTAINERS`.** If you run stacks that manage their own updates (Nextcloud AIO, etc.), list those containers here (space-separated) to prevent Watchtower from interfering.
+
+## See Also
+
+- [docker-healthchecks](docker-healthchecks.md)
+- [debugging-broken-docker-containers](debugging-broken-docker-containers.md)