majorwiki/02-selfhosting/monitoring/logwatch-fleet-setup.md

---
title: Logwatch Fleet Setup — Surviving Package Upgrades
description: Configure logwatch on mixed Debian/Fedora fleets so settings survive package upgrades
tags:
  - logwatch
  - monitoring
  - ansible
  - fedora
  - ubuntu
status: published
created: 2026-05-09
updated: 2026-05-10T13:00
---

# Logwatch Fleet Setup — Surviving Package Upgrades

Logwatch ships with a defaults file at `/usr/share/logwatch/default.conf/logwatch.conf`. On Fedora, package upgrades **silently reset** this file — wiping any customizations. The fix is to put all settings in the **local override file** at `/etc/logwatch/conf/logwatch.conf`, which is never touched by package managers.

## The Problem

Fedora 44's logwatch 7.14-1 upgrade (April 2026) reset `Output` from `mail` back to `stdout` in the defaults file. Servers that had been emailing daily reports for months went silent with zero errors. `rpm -V logwatch` shows the defaults file was modified (`S.5....T.`), but there's no warning during upgrade.

Ubuntu is less affected because its `/etc/cron.daily/00logwatch` script passes `--output mail` explicitly, overriding the config. Fedora's cron script does not.

## The Fix

Write all settings to the **override file** (`/etc/logwatch/conf/logwatch.conf`):

```ini
# Managed by Ansible — do not edit manually.
# Local overrides — survives package upgrades.
Output = mail
MailTo = marcus@majorshouse.com
MailFrom = Logwatch@hostname.majorshouse.com
Detail = Low
```

Key settings:

| Setting | Value | Why |
|---------|-------|-----|
| `Output` | `mail` | Must be `mail`, not `stdout`. Fedora's cron script doesn't pass `--output mail` like Ubuntu's does. |
| `MailTo` | recipient address | Where reports go. |
| `MailFrom` | per-host sender | Makes it easy to identify which server sent the report. |
| `Detail` | `Low` | Keeps emails scannable. Raise to `Med` or `High` for debugging. |

## Ansible Playbook

The `logwatch.yml` playbook handles both OS families:

```yaml
- name: Install and configure logwatch
  hosts: all
  become: true
  gather_facts: true
  tasks:
    - name: Install logwatch (Debian/Ubuntu)
      ansible.builtin.apt:
        name: logwatch
        state: present
      when: ansible_facts['os_family'] == "Debian"

    - name: Install logwatch (Fedora)
      ansible.builtin.dnf:
        name: logwatch
        state: present
      when: ansible_facts['os_family'] == "RedHat"

    - name: Ensure logwatch override directory exists
      ansible.builtin.file:
        path: /etc/logwatch/conf
        state: directory
        mode: '0755'

    - name: Configure logwatch override (survives package upgrades)
      ansible.builtin.copy:
        dest: /etc/logwatch/conf/logwatch.conf
        mode: '0644'
        content: |
          # Managed by Ansible — do not edit manually.
          Output = mail
          MailTo = {{ logwatch_email }}
          MailFrom = Logwatch@{{ inventory_hostname }}.majorshouse.com
          Detail = Low
```

Include it in `harden.yml` so every new server gets logwatch as part of the baseline.

## Verifying

After deploying, test immediately:

```bash
sudo logwatch --output mail --range today
```

Check that the email arrives. If it doesn't, verify Postfix is installed and relaying correctly — logwatch depends on a working local MTA.

## Diagnosing Silent Failures

```bash
# Check if the defaults file was modified by a package upgrade
rpm -V logwatch  # Fedora
dpkg -V logwatch  # Debian

# Look for S.5....T. on the defaults file — means it was replaced
# S = size, 5 = md5, T = timestamp changed
```

## Fedora CA Bundle Missing — Postfix TLS Engine Unavailable

If the Fedora half of your fleet is silent but the Debian/Ubuntu half is fine, and your relayhost requires TLS, suspect a missing CA bundle. Symptom on the sending host:

```
postfix/error: status=deferred (delivery temporarily suspended:
TLS is required, but our TLS engine is unavailable)
```

The tell that this is the CA bundle and not a postfix-internal problem: **dnf and curl are also broken on the box.** Run any `sudo dnf list` / `sudo curl https://...` and look for:

```
Curl error (77): Problem with the SSL CA cert (path? access rights?)
[error adding trust anchors from file: /etc/pki/tls/certs/ca-bundle.crt]
```

That's the same path postfix's `smtp_tls_CAfile` defaults to. Every TLS client on the box is failing because a single symlink is missing.

### Diagnosis

```bash
# Is the consumer-path symlink there?
ls -la /etc/pki/tls/certs/ca-bundle.crt
# Expected: lrwxrwxrwx ... -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

# Is the extracted bundle itself intact?
ls -la /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
sudo grep -c 'BEGIN CERTIFICATE' /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
# Expected: ~140-150 certs, ~220 KB
```

If the extracted bundle exists but the consumer-path symlink is gone, you've found it. `update-ca-trust extract` regenerates the `extracted/` paths but does **not** recreate the upstream-style symlink at `/etc/pki/tls/certs/ca-bundle.crt` — that symlink is shipped by the `ca-certificates` package and can be lost during a partial upgrade or a stray `rm`.

### Fix

```bash
sudo ln -sfn /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
            /etc/pki/tls/certs/ca-bundle.crt
sudo systemctl reload postfix
sudo postqueue -f       # drain deferred mail
```

Verify with `sudo grep -c 'BEGIN CERTIFICATE' /etc/pki/tls/certs/ca-bundle.crt` (should match the extracted bundle's count) and `sudo dnf list --installed postfix` (should no longer show the curl error).

### Audit the rest of the Fedora fleet

Once you find one host with this issue, check the others — package events that broke one box may have broken its siblings:

```bash
for host in $(your fleet | grep fedora); do
  echo "$host: $(ssh $host 'ls /etc/pki/tls/certs/ca-bundle.crt 2>&1' | tail -1)"
done
```

Hosts returning "No such file or directory" are silently broken. They won't fail loudly until something asks them to do TLS — which on a small homelab might be never until logwatch tries to mail you weeks later.

### Methodology note: postfix logs differ between distros

Don't trust a single log source when surveying a mixed fleet. **Fedora and majormail log postfix to journald** (`journalctl -u postfix`); **Debian/Ubuntu log to `/var/log/mail.log`** (and rotated `mail.log.1` / `mail.log.*.gz`). Querying journalctl on Ubuntu returns "no entries" even when mail is flowing — easy way to declare a working host broken. Always run `tail /var/log/mail.log` on Debian-family hosts and `journalctl -u postfix` on Fedora-family hosts.

## Bounce-source addresses must be real mailboxes

A subtle related class of bug: services like Watchtower, fail2ban, cron, and Netdata default to sending notifications **from** an identity that doesn't exist as a recipient — `watchtower@majorshouse.com`, `fail2ban@<host>.majorshouse.com`, `root@<host>.localdomain`. While the relayhost is healthy, nobody notices. The moment any delivery fails (network blip, recipient typo, queue overflow, the CA bundle bug above), the local MTA tries to bounce the original message back to that sender — finds no mailbox — and the bounce itself bounces. You get MAILER-DAEMON queue churn and `5.7.1 Relay access denied` rejections in your mail server logs.

Fix it once at the source: set `WATCHTOWER_NOTIFICATION_EMAIL_FROM`, fail2ban's `sender =`, and similar to a **real mailbox** on your mail server (e.g., `marcus@majorshouse.com`). Bounces then land somewhere a human can read them, and the noise disappears.

## Per-host config drift on cloud-image-derived servers

When fleet hosts are spun up from images (DigitalOcean droplet snapshots, Packer artifacts, cloud-init templates), three specific config drift patterns silently break notification mail. Each one looks fine in isolation; the combination produces "mail leaves the host with `250 OK queued` and disappears."

### 1. Packer/snapshot-leftover `myhostname` in postfix

A host built from a Packer-baked image often has `postfix myhostname = packer-<uuid>` baked into `main.cf` from the build process. The system hostname might have been correctly set by terraform/cloud-init at first boot, but postfix's `myhostname` was hardcoded during image build and was never overridden. Result: every outbound message-id and EHLO carries the Packer artifact name (e.g., `<20260509120011.7EB6ABD83C@packer-641079bc-bc17-b5e1-1425-be745d012d0b>`), no SPF/DKIM matches that name, and remote spam filters score it as suspicious.

**Detect:**

```bash
postconf myhostname | grep -E 'packer-|builder-|<image-build-prefix>'
```

**Fix:**

```bash
hostnamectl set-hostname <real-fqdn>
postconf -e 'myhostname = <real-fqdn>'
sed -i '/^127\.0\.1\.1/d' /etc/hosts && \
  echo "127.0.1.1 <real-fqdn> <short-name>" >> /etc/hosts
systemctl reload postfix
```

### 2. Empty `relayhost` quietly forces public-MX delivery

If `postconf relayhost` returns an empty value, postfix doesn't fail — it just does an MX lookup for the destination domain and tries to deliver directly. For mail to your own mail server, that means going via the **public MX** (the domain's external MX record, e.g., `mail.majorshouse.com → 165.227.187.191:25`) instead of the **internal/Tailscale relay path** the rest of the fleet uses.

The public-MX path is subject to whatever spam filtering, content checks, and trust rules the receiving MX has configured for external traffic. Internal Tailscale-IP traffic typically gets a faster trust shortcut (e.g., bypass spamchk pipe). So this single configuration drift causes one host's mail to land in a different code path than its siblings — and then silently get filtered.

**Detect:** look for fleet hosts where `postconf relayhost` returns blank and compare to known-good siblings.

**Fix:** set `relayhost = [<mailserver-tailscale-ip>]:587` (or whatever port your fleet convention uses).

### 3. Stale SASL passwd map referencing a missing file

Postfix configurations migrated from a previous setup often retain `smtp_sasl_auth_enable = yes` and `smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd` even when no SASL is needed for the current relay path. If the actual `sasl_passwd` file isn't there (because the migration didn't carry it, or the new relay doesn't require auth), every send attempt produces:

```
error: open database /etc/postfix/sasl_passwd.db: No such file or directory
warning: smtp_sasl_password_maps lookup error
status=deferred (local data error while talking to <relay>)
```

Especially common after migrating from external SMTP (SendGrid, Mailgun, etc., which use SASL) to an internal Tailscale relay (which doesn't).

**Detect:**

```bash
postconf -n | grep -E 'smtp_sasl_(auth_enable|password_maps)'
[ -f /etc/postfix/sasl_passwd ] || echo "sasl_passwd file missing"
```

**Fix — disable SASL if the new relay doesn't need it:**

```bash
postconf -e 'smtp_sasl_auth_enable = no'
postconf -e 'smtp_tls_wrappermode = no'   # if switching from port 465 to 587
postconf -X 'smtp_sasl_password_maps'
systemctl reload postfix
```

### Audit shortcut

For a quick per-host comparison across the fleet:

```bash
for host in your fleet hosts; do
  echo "=== $host ==="
  ssh "$host" 'postconf myhostname relayhost smtp_sasl_auth_enable 2>&1' | head -3
done
```

Anomalies (Packer hostnames, blank relayhost, SASL enabled where siblings have it disabled) jump out immediately.

## Lesson Learned

Never customize `/usr/share/logwatch/default.conf/logwatch.conf`. Always use `/etc/logwatch/conf/logwatch.conf`. This applies to any software that has a "defaults" file and an "override" file — the override survives upgrades, the defaults file does not.

A second, broader lesson from the 2026-05-10 fleet outage: **silent fleet-wide email gaps are usually a stack of unrelated failures, not one cause.** That morning's investigation surfaced a missing CA bundle on two Fedora hosts, a postfix relayhost using a name that postfix's resolver couldn't handle, two services with non-mailbox sender addresses generating bounce churn, and a corrupt syslog-vs-journald assumption that hid working hosts. Each was minor in isolation. Together they made all seven hosts look broken when in fact only two were. Triage by ground-truth (what arrived in the destination mailbox) before assuming what's broken at the source.