majorwiki/05-troubleshooting/logwatch-wrong-hostname-after-migration.md
majorlinux 852375ddf0 logwatch-hostname wiki: add hostname-correct-but-config-baked variant
majormail (2026-06-14) had the correct system hostname but still mailed
from majormail-hetzner — the old provisioning label was hardcoded in
logwatch.conf MailFrom and fail2ban jail.local sender. Add a variant
section covering the config grep sweep and the templated-vs-static
Ansible regression caveat.
2026-06-14 04:00:18 -04:00

150 lines
6.2 KiB
Markdown

---
title: "Logwatch Reports the Wrong Hostname (`<host>-hetzner`) After a Migration"
domain: troubleshooting
category: monitoring
tags: [logwatch, hostname, hetzner, migration, monitoring, provisioning, fail2ban]
status: published
created: 2026-06-12
updated: 2026-06-14
---
# Logwatch Reports the Wrong Hostname (`<host>-hetzner`) After a Migration
## Symptom
Daily Logwatch emails from a recently migrated server arrive titled with the
provisioning label instead of the real hostname:
```
Logwatch for tttpod-hetzner (Linux)
Logwatch for dcaprod-hetzner (Linux)
```
Everything else works — the report is generated, mailed, and delivered. Only the
**name in the title is wrong**, which makes reports harder to scan and breaks any
filter or rule that keys on the expected hostname.
## Cause
Logwatch titles each report with the box's **live system hostname**
(`hostnamectl --static` / `/etc/hostname`) read at runtime — it does *not* keep
its own copy of the name.
Hetzner Cloud servers are provisioned with a temporary node label as the system
hostname — `<host>-hetzner` (e.g. `tttpod-hetzner`). The migration runbook renames
the **Tailscale node** back to the bare name and sets Postfix `myhostname`, but the
**OS hostname** itself is easy to miss because nothing surfaces it day to day. It
stays `<host>-hetzner` until something reads `hostname` — Logwatch is usually the
first thing to do so, weeks later.
Confirm the box is actually mislabelled:
```bash
ssh root@<host> 'hostnamectl --static; cat /etc/hostname; grep 127.0.1.1 /etc/hosts'
# static: tttpod-hetzner
# /etc/hostname: tttpod-hetzner
# 127.0.1.1 tttpod-hetzner tttpod-hetzner
```
## Fix
Set the real hostname and fix the matching `/etc/hosts` loopback line:
```bash
ssh root@<host> '
hostnamectl set-hostname <host>
sed -i "s/127.0.1.1.*/127.0.1.1 <host> <host>/" /etc/hosts
hostnamectl --static # verify -> <host>
'
```
That's it. **Logwatch has no hardcoded hostname override** — verify with:
```bash
grep -ri hostname /etc/logwatch/ /etc/cron.daily/0logwatch /etc/cron.daily/logwatch 2>/dev/null
cat /etc/mailname 2>/dev/null
```
If those are empty (the normal case), Logwatch reads the live hostname on its next
run, so the **next daily report self-corrects** — no service restart, no logwatch
config change needed.
> [!note] If `grep` *does* find a hostname pinned in `/etc/logwatch/conf/logwatch.conf`
> (e.g. a `HostLimit`/`MailFrom` line baked in by Ansible), update it there too —
> the override file wins over the live hostname.
## Sweep the whole fleet
This is a per-box provisioning leftover, so check every migrated host at once —
more than one is usually affected:
```bash
for ip in 100.98.223.93 100.95.137.38 100.64.169.62 100.112.127.0 100.73.85.46; do
echo -n "$ip -> "
ssh -o ConnectTimeout=8 -o BatchMode=yes root@$ip 'hostnamectl --static' 2>/dev/null \
|| echo '(unreachable)'
done
```
Any value ending in `-hetzner` (or your provider's build label) needs the fix above.
In the 2026-06 sweep, `tttpod` and `dcaprod` were still `*-hetzner` at the OS
level; `majortoot`, `majormail`, and `majorlinux` had the correct system hostname
— but see the variant below: `majormail`'s *configs* were still stale even though
its hostname wasn't.
## Variant: hostname is correct, but a config has the old name baked in
A second, sneakier form of this drift: the **system hostname is already right**, so
the sweep above passes and the Logwatch report *title* is correct — yet mail still
arrives **from** `<host>-hetzner` because the old label is hardcoded in a service's
`From`/`sender` field. These fields are static text, not derived from the live
hostname, so fixing `hostnamectl` does nothing for them.
Seen on `majormail` (2026-06-14): system hostname was `majormail`, but
`Logwatch@majormail-hetzner...` was still the sender. Two configs held it:
```bash
# sweep a box for the old provisioning label in any send-related config
ssh root@<host> 'grep -rsn "<host>-hetzner" /etc/logwatch/ /etc/fail2ban/ \
/etc/postfix/ /etc/aliases /etc/mailname 2>/dev/null'
# /etc/logwatch/conf/logwatch.conf:MailFrom = Logwatch@<host>-hetzner.majorshouse.com
# /etc/fail2ban/jail.local:sender = fail2ban@<host>-hetzner.majorshouse.com
```
Fix in place (no restart needed for Logwatch; reload fail2ban for its change):
```bash
ssh root@<host> '
sed -i "s/<host>-hetzner/<host>/g" /etc/logwatch/conf/logwatch.conf /etc/fail2ban/jail.local
systemctl reload fail2ban
'
```
> [!warning] Check the Ansible source, or it comes back
> A live `sed` is undone by the next playbook run if the repo still carries the old
> value. Distinguish two cases:
> - **Templated** (safe): e.g. `logwatch.yml` sets `MailFrom = Logwatch@{{ inventory_hostname }}...`. If the inventory host is named correctly, a run *regenerates* the right value — it even self-heals a stale box.
> - **Static file** (will regress): e.g. `roles/fail2ban/files/hosts/<host>/jail.local` with the literal `sender = ...@<host>-hetzner...`. Grep the repo (`grep -rn "<host>-hetzner" .`) and fix the file too, or every deploy re-pushes the stale sender.
Inert backups (`jail.local.bak*`, `*~`) may still contain the old string — they
don't send mail, so leave them.
## Prevention
Fold "set the system hostname" into the migration bootstrap so it never drifts:
```bash
hostnamectl set-hostname <host>
sed -i "s/127.0.1.1.*/127.0.1.1 <host> <host>/" /etc/hosts
```
Do this in the **same step** that renames the Tailscale node and sets Postfix
`myhostname` — all three read from the provisioning label and all three must be
corrected together. See the
[VPS Migration Baseline Checklist](../02-selfhosting/cloud/vps-migration-baseline-checklist.md).
## Related
- [Logwatch Fleet Setup — Surviving Package Upgrades](../02-selfhosting/monitoring/logwatch-fleet-setup.md) — the broader "logwatch went silent / wrong-source" class, including the Packer `myhostname` variant of this same drift
- [VPS Migration Baseline Checklist](../02-selfhosting/cloud/vps-migration-baseline-checklist.md) — the full post-migration verification list
- [Ansible UNREACHABLE: Host Key Verification Failed After a Host Rebuild or Migration](networking/ansible-host-key-verification-failed-rebuilt-host.md) — another IP/identity-drift gotcha from the same Hetzner migration