majormail (2026-06-14) had the correct system hostname but still mailed from majormail-hetzner — the old provisioning label was hardcoded in logwatch.conf MailFrom and fail2ban jail.local sender. Add a variant section covering the config grep sweep and the templated-vs-static Ansible regression caveat.
150 lines
6.2 KiB
Markdown
150 lines
6.2 KiB
Markdown
---
|
|
title: "Logwatch Reports the Wrong Hostname (`<host>-hetzner`) After a Migration"
|
|
domain: troubleshooting
|
|
category: monitoring
|
|
tags: [logwatch, hostname, hetzner, migration, monitoring, provisioning, fail2ban]
|
|
status: published
|
|
created: 2026-06-12
|
|
updated: 2026-06-14
|
|
---
|
|
|
|
# Logwatch Reports the Wrong Hostname (`<host>-hetzner`) After a Migration
|
|
|
|
## Symptom
|
|
|
|
Daily Logwatch emails from a recently migrated server arrive titled with the
|
|
provisioning label instead of the real hostname:
|
|
|
|
```
|
|
Logwatch for tttpod-hetzner (Linux)
|
|
Logwatch for dcaprod-hetzner (Linux)
|
|
```
|
|
|
|
Everything else works — the report is generated, mailed, and delivered. Only the
|
|
**name in the title is wrong**, which makes reports harder to scan and breaks any
|
|
filter or rule that keys on the expected hostname.
|
|
|
|
## Cause
|
|
|
|
Logwatch titles each report with the box's **live system hostname**
|
|
(`hostnamectl --static` / `/etc/hostname`) read at runtime — it does *not* keep
|
|
its own copy of the name.
|
|
|
|
Hetzner Cloud servers are provisioned with a temporary node label as the system
|
|
hostname — `<host>-hetzner` (e.g. `tttpod-hetzner`). The migration runbook renames
|
|
the **Tailscale node** back to the bare name and sets Postfix `myhostname`, but the
|
|
**OS hostname** itself is easy to miss because nothing surfaces it day to day. It
|
|
stays `<host>-hetzner` until something reads `hostname` — Logwatch is usually the
|
|
first thing to do so, weeks later.
|
|
|
|
Confirm the box is actually mislabelled:
|
|
|
|
```bash
|
|
ssh root@<host> 'hostnamectl --static; cat /etc/hostname; grep 127.0.1.1 /etc/hosts'
|
|
# static: tttpod-hetzner
|
|
# /etc/hostname: tttpod-hetzner
|
|
# 127.0.1.1 tttpod-hetzner tttpod-hetzner
|
|
```
|
|
|
|
## Fix
|
|
|
|
Set the real hostname and fix the matching `/etc/hosts` loopback line:
|
|
|
|
```bash
|
|
ssh root@<host> '
|
|
hostnamectl set-hostname <host>
|
|
sed -i "s/127.0.1.1.*/127.0.1.1 <host> <host>/" /etc/hosts
|
|
hostnamectl --static # verify -> <host>
|
|
'
|
|
```
|
|
|
|
That's it. **Logwatch has no hardcoded hostname override** — verify with:
|
|
|
|
```bash
|
|
grep -ri hostname /etc/logwatch/ /etc/cron.daily/0logwatch /etc/cron.daily/logwatch 2>/dev/null
|
|
cat /etc/mailname 2>/dev/null
|
|
```
|
|
|
|
If those are empty (the normal case), Logwatch reads the live hostname on its next
|
|
run, so the **next daily report self-corrects** — no service restart, no logwatch
|
|
config change needed.
|
|
|
|
> [!note] If `grep` *does* find a hostname pinned in `/etc/logwatch/conf/logwatch.conf`
|
|
> (e.g. a `HostLimit`/`MailFrom` line baked in by Ansible), update it there too —
|
|
> the override file wins over the live hostname.
|
|
|
|
## Sweep the whole fleet
|
|
|
|
This is a per-box provisioning leftover, so check every migrated host at once —
|
|
more than one is usually affected:
|
|
|
|
```bash
|
|
for ip in 100.98.223.93 100.95.137.38 100.64.169.62 100.112.127.0 100.73.85.46; do
|
|
echo -n "$ip -> "
|
|
ssh -o ConnectTimeout=8 -o BatchMode=yes root@$ip 'hostnamectl --static' 2>/dev/null \
|
|
|| echo '(unreachable)'
|
|
done
|
|
```
|
|
|
|
Any value ending in `-hetzner` (or your provider's build label) needs the fix above.
|
|
In the 2026-06 sweep, `tttpod` and `dcaprod` were still `*-hetzner` at the OS
|
|
level; `majortoot`, `majormail`, and `majorlinux` had the correct system hostname
|
|
— but see the variant below: `majormail`'s *configs* were still stale even though
|
|
its hostname wasn't.
|
|
|
|
## Variant: hostname is correct, but a config has the old name baked in
|
|
|
|
A second, sneakier form of this drift: the **system hostname is already right**, so
|
|
the sweep above passes and the Logwatch report *title* is correct — yet mail still
|
|
arrives **from** `<host>-hetzner` because the old label is hardcoded in a service's
|
|
`From`/`sender` field. These fields are static text, not derived from the live
|
|
hostname, so fixing `hostnamectl` does nothing for them.
|
|
|
|
Seen on `majormail` (2026-06-14): system hostname was `majormail`, but
|
|
`Logwatch@majormail-hetzner...` was still the sender. Two configs held it:
|
|
|
|
```bash
|
|
# sweep a box for the old provisioning label in any send-related config
|
|
ssh root@<host> 'grep -rsn "<host>-hetzner" /etc/logwatch/ /etc/fail2ban/ \
|
|
/etc/postfix/ /etc/aliases /etc/mailname 2>/dev/null'
|
|
# /etc/logwatch/conf/logwatch.conf:MailFrom = Logwatch@<host>-hetzner.majorshouse.com
|
|
# /etc/fail2ban/jail.local:sender = fail2ban@<host>-hetzner.majorshouse.com
|
|
```
|
|
|
|
Fix in place (no restart needed for Logwatch; reload fail2ban for its change):
|
|
|
|
```bash
|
|
ssh root@<host> '
|
|
sed -i "s/<host>-hetzner/<host>/g" /etc/logwatch/conf/logwatch.conf /etc/fail2ban/jail.local
|
|
systemctl reload fail2ban
|
|
'
|
|
```
|
|
|
|
> [!warning] Check the Ansible source, or it comes back
|
|
> A live `sed` is undone by the next playbook run if the repo still carries the old
|
|
> value. Distinguish two cases:
|
|
> - **Templated** (safe): e.g. `logwatch.yml` sets `MailFrom = Logwatch@{{ inventory_hostname }}...`. If the inventory host is named correctly, a run *regenerates* the right value — it even self-heals a stale box.
|
|
> - **Static file** (will regress): e.g. `roles/fail2ban/files/hosts/<host>/jail.local` with the literal `sender = ...@<host>-hetzner...`. Grep the repo (`grep -rn "<host>-hetzner" .`) and fix the file too, or every deploy re-pushes the stale sender.
|
|
|
|
Inert backups (`jail.local.bak*`, `*~`) may still contain the old string — they
|
|
don't send mail, so leave them.
|
|
|
|
## Prevention
|
|
|
|
Fold "set the system hostname" into the migration bootstrap so it never drifts:
|
|
|
|
```bash
|
|
hostnamectl set-hostname <host>
|
|
sed -i "s/127.0.1.1.*/127.0.1.1 <host> <host>/" /etc/hosts
|
|
```
|
|
|
|
Do this in the **same step** that renames the Tailscale node and sets Postfix
|
|
`myhostname` — all three read from the provisioning label and all three must be
|
|
corrected together. See the
|
|
[VPS Migration Baseline Checklist](../02-selfhosting/cloud/vps-migration-baseline-checklist.md).
|
|
|
|
## Related
|
|
|
|
- [Logwatch Fleet Setup — Surviving Package Upgrades](../02-selfhosting/monitoring/logwatch-fleet-setup.md) — the broader "logwatch went silent / wrong-source" class, including the Packer `myhostname` variant of this same drift
|
|
- [VPS Migration Baseline Checklist](../02-selfhosting/cloud/vps-migration-baseline-checklist.md) — the full post-migration verification list
|
|
- [Ansible UNREACHABLE: Host Key Verification Failed After a Host Rebuild or Migration](networking/ansible-host-key-verification-failed-rebuilt-host.md) — another IP/identity-drift gotcha from the same Hetzner migration
|