wiki: add Fedora CA bundle article, update migration checklist and logwatch docs
New article documenting missing /etc/pki/tls/certs/ca-bundle.crt symlink on Hetzner Fedora images breaking Postfix TLS, curl, and dnf. Updated VPS migration baseline checklist with timezone, CA bundle, and crond verification steps. Updated logwatch fleet setup with crond check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
488268ccd1
commit
de9b661b9d
4 changed files with 492 additions and 1 deletions
96
02-selfhosting/cloud/vps-migration-baseline-checklist.md
Normal file
96
02-selfhosting/cloud/vps-migration-baseline-checklist.md
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
---
|
||||
title: VPS Migration Baseline Checklist
|
||||
description: What to verify after migrating a server to a new provider — the packages, services, and configs that must match the old box
|
||||
tags:
|
||||
- migration
|
||||
- vps
|
||||
- hetzner
|
||||
- digitalocean
|
||||
- ansible
|
||||
- checklist
|
||||
status: published
|
||||
created: 2026-05-09
|
||||
updated: 2026-05-11T07:33
|
||||
---
|
||||
|
||||
# VPS Migration Baseline Checklist
|
||||
|
||||
When migrating a server from one VPS provider to another, it's easy to focus on the application (bots, web services, databases) and forget the infrastructure baseline. This checklist covers the common components that make a server operational beyond just running the app.
|
||||
|
||||
## Background
|
||||
|
||||
During the Hetzner migration (2026-05), `majordiscord` was migrated with only the application layer (PhantomBot, Red-DiscordBot) and core infrastructure (Netdata, Tailscale, fail2ban). Missing from the new box: Postfix (email relay), logwatch, ClamAV, and dnf-automatic. The gap went unnoticed for a week because all monitoring email depended on the missing Postfix.
|
||||
|
||||
## The Checklist
|
||||
|
||||
### Before Migration
|
||||
|
||||
Power on both old and new boxes. Run this comparison to find gaps:
|
||||
|
||||
```bash
|
||||
# Fedora — list baseline packages on both hosts
|
||||
ssh root@OLD_HOST 'rpm -qa --qf "%{NAME}\n" | sort | grep -iE "fail2ban|logwatch|postfix|netdata|clamav|dnf-auto|tailscale|cronie|firewalld"'
|
||||
ssh root@NEW_HOST 'rpm -qa --qf "%{NAME}\n" | sort | grep -iE "fail2ban|logwatch|postfix|netdata|clamav|dnf-auto|tailscale|cronie|firewalld"'
|
||||
|
||||
# Ubuntu — list baseline packages on both hosts
|
||||
ssh root@OLD_HOST 'dpkg -l | grep -iE "fail2ban|logwatch|postfix|netdata|clamav|unattended|tailscale" | awk "{print \$2}" | sort'
|
||||
ssh root@NEW_HOST 'dpkg -l | grep -iE "fail2ban|logwatch|postfix|netdata|clamav|unattended|tailscale" | awk "{print \$2}" | sort'
|
||||
```
|
||||
|
||||
Compare enabled services:
|
||||
|
||||
```bash
|
||||
ssh root@HOST 'systemctl list-unit-files --state=enabled --no-pager | grep -iE "fail2ban|logwatch|postfix|netdata|clamav|dnf-auto|tailscale|cronie|firewalld|sshd"'
|
||||
```
|
||||
|
||||
### Baseline Components
|
||||
|
||||
Every server in the fleet should have these. Check each one after migration:
|
||||
|
||||
| Component | Package (Fedora) | Package (Ubuntu) | Ansible Playbook | Notes |
|
||||
|-----------|-----------------|------------------|------------------|-------|
|
||||
| Monitoring | `netdata` | `netdata` | `netdata.yml` | Claim to Netdata Cloud if applicable |
|
||||
| VPN | `tailscale` | `tailscale` | — (manual join) | Rename node in Tailscale admin |
|
||||
| Intrusion prevention | `fail2ban` | `fail2ban` | `harden.yml` | Check jail.local, banaction matches firewall |
|
||||
| Email relay | `postfix` | `postfix` | `configure_postfix_relay.yml` | Required by logwatch, Netdata, fail2ban |
|
||||
| Log summaries | `logwatch` | `logwatch` | `logwatch.yml` | Override file, not defaults — see [logwatch fleet setup](../monitoring/logwatch-fleet-setup.md) |
|
||||
| Firewall | `firewalld` | `ufw` | `configure_firewall_*.yml` | Verify fail2ban banaction matches |
|
||||
| Cron | `cronie` | `cron` | — (usually pre-installed) | Required by logwatch |
|
||||
| Auto-updates | `dnf-automatic` | `unattended-upgrades` | `ansible-unattended-upgrades-fleet` | Security patches only |
|
||||
| Antivirus | `clamav` | `clamav` | `configure_clamav.yml` | Internet-facing hosts only |
|
||||
| SSH hardening | `openssh-server` | `openssh-server` | `configure_ssh_hardening.yml` | Key-only, no root password |
|
||||
| Timezone | — | — | — | US servers: `America/New_York`; UK: `Europe/London`. Hetzner defaults to UTC. |
|
||||
| CA bundle (Fedora) | `ca-certificates` | `ca-certificates` | — | Verify `/etc/pki/tls/certs/ca-bundle.crt` symlink exists — see [Fedora CA bundle fix](../../05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md) |
|
||||
|
||||
### After Migration
|
||||
|
||||
1. **Set the timezone** — `timedatectl set-timezone America/New_York` (US) or `Europe/London` (UK). Hetzner images default to UTC.
|
||||
2. **Verify CA bundle (Fedora)** — `ls /etc/pki/tls/certs/ca-bundle.crt`. If missing, Postfix TLS, curl, and dnf will all fail silently. See [Fedora CA bundle fix](../../05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md).
|
||||
3. **Run `harden.yml` against the new host** — catches most gaps in one pass
|
||||
4. **Send a test email** — `echo test | mail -s "test" marcus@majorshouse.com` — if this fails, nothing else can alert you
|
||||
5. **Verify crond is running** — `systemctl is-active crond` (Fedora) or `systemctl is-active cron` (Ubuntu). cronie can be `enabled` but not `active` after provisioning.
|
||||
6. **Check Netdata Cloud** — verify the new node appears and alerts are flowing
|
||||
7. **Compare fail2ban jails** — `fail2ban-client status` on both old and new
|
||||
8. **Verify logwatch sends** — `sudo logwatch --output mail --range today`
|
||||
9. **Keep the old box powered off but not destroyed** for at least 7 days after remediation
|
||||
|
||||
### Using doctl to Manage Old Droplets
|
||||
|
||||
```bash
|
||||
# Authenticate (token from Ansible vault)
|
||||
cd ~/MajorAnsible
|
||||
ansible-vault view group_vars/all/vault.yml | grep vault_do_oauth_token | awk '{print $2}' | xargs doctl auth init --access-token
|
||||
|
||||
# List droplets
|
||||
doctl compute droplet list --format Name,ID,Status,PublicIPv4
|
||||
|
||||
# Power on for comparison
|
||||
doctl compute droplet-action power-on DROPLET_ID
|
||||
|
||||
# Power off when done
|
||||
doctl compute droplet-action power-off DROPLET_ID
|
||||
```
|
||||
|
||||
## Lesson Learned
|
||||
|
||||
Application migration is not server migration. The app can work perfectly while the monitoring, alerting, and email infrastructure is completely broken. Always compare the full package baseline between old and new boxes before calling a migration complete.
|
||||
267
02-selfhosting/monitoring/logwatch-fleet-setup.md
Normal file
267
02-selfhosting/monitoring/logwatch-fleet-setup.md
Normal file
|
|
@ -0,0 +1,267 @@
|
|||
---
|
||||
title: Logwatch Fleet Setup — Surviving Package Upgrades
|
||||
description: Configure logwatch on mixed Debian/Fedora fleets so settings survive package upgrades
|
||||
tags:
|
||||
- logwatch
|
||||
- monitoring
|
||||
- ansible
|
||||
- fedora
|
||||
- ubuntu
|
||||
status: published
|
||||
created: 2026-05-09
|
||||
updated: 2026-05-11T07:33
|
||||
---
|
||||
|
||||
# Logwatch Fleet Setup — Surviving Package Upgrades
|
||||
|
||||
Logwatch ships with a defaults file at `/usr/share/logwatch/default.conf/logwatch.conf`. On Fedora, package upgrades **silently reset** this file — wiping any customizations. The fix is to put all settings in the **local override file** at `/etc/logwatch/conf/logwatch.conf`, which is never touched by package managers.
|
||||
|
||||
## The Problem
|
||||
|
||||
Fedora 44's logwatch 7.14-1 upgrade (April 2026) reset `Output` from `mail` back to `stdout` in the defaults file. Servers that had been emailing daily reports for months went silent with zero errors. `rpm -V logwatch` shows the defaults file was modified (`S.5....T.`), but there's no warning during upgrade.
|
||||
|
||||
Ubuntu is less affected because its `/etc/cron.daily/00logwatch` script passes `--output mail` explicitly, overriding the config. Fedora's cron script does not.
|
||||
|
||||
## The Fix
|
||||
|
||||
Write all settings to the **override file** (`/etc/logwatch/conf/logwatch.conf`):
|
||||
|
||||
```ini
|
||||
# Managed by Ansible — do not edit manually.
|
||||
# Local overrides — survives package upgrades.
|
||||
Output = mail
|
||||
MailTo = marcus@majorshouse.com
|
||||
MailFrom = Logwatch@hostname.majorshouse.com
|
||||
Detail = Low
|
||||
```
|
||||
|
||||
Key settings:
|
||||
|
||||
| Setting | Value | Why |
|
||||
|---------|-------|-----|
|
||||
| `Output` | `mail` | Must be `mail`, not `stdout`. Fedora's cron script doesn't pass `--output mail` like Ubuntu's does. |
|
||||
| `MailTo` | recipient address | Where reports go. |
|
||||
| `MailFrom` | per-host sender | Makes it easy to identify which server sent the report. |
|
||||
| `Detail` | `Low` | Keeps emails scannable. Raise to `Med` or `High` for debugging. |
|
||||
|
||||
## Ansible Playbook
|
||||
|
||||
The `logwatch.yml` playbook handles both OS families:
|
||||
|
||||
```yaml
|
||||
- name: Install and configure logwatch
|
||||
hosts: all
|
||||
become: true
|
||||
gather_facts: true
|
||||
tasks:
|
||||
- name: Install logwatch (Debian/Ubuntu)
|
||||
ansible.builtin.apt:
|
||||
name: logwatch
|
||||
state: present
|
||||
when: ansible_facts['os_family'] == "Debian"
|
||||
|
||||
- name: Install logwatch (Fedora)
|
||||
ansible.builtin.dnf:
|
||||
name: logwatch
|
||||
state: present
|
||||
when: ansible_facts['os_family'] == "RedHat"
|
||||
|
||||
- name: Ensure logwatch override directory exists
|
||||
ansible.builtin.file:
|
||||
path: /etc/logwatch/conf
|
||||
state: directory
|
||||
mode: '0755'
|
||||
|
||||
- name: Configure logwatch override (survives package upgrades)
|
||||
ansible.builtin.copy:
|
||||
dest: /etc/logwatch/conf/logwatch.conf
|
||||
mode: '0644'
|
||||
content: |
|
||||
# Managed by Ansible — do not edit manually.
|
||||
Output = mail
|
||||
MailTo = {{ logwatch_email }}
|
||||
MailFrom = Logwatch@{{ inventory_hostname }}.majorshouse.com
|
||||
Detail = Low
|
||||
```
|
||||
|
||||
Include it in `harden.yml` so every new server gets logwatch as part of the baseline.
|
||||
|
||||
## Verifying
|
||||
|
||||
After deploying, test immediately:
|
||||
|
||||
```bash
|
||||
# Verify crond is actually running — cronie can be "enabled" but not "active"
|
||||
systemctl is-active crond # Fedora
|
||||
systemctl is-active cron # Ubuntu
|
||||
|
||||
# If inactive, start it
|
||||
sudo systemctl start crond
|
||||
|
||||
# Then test logwatch manually
|
||||
sudo logwatch --output mail --range today
|
||||
```
|
||||
|
||||
Check that the email arrives. If it doesn't, verify:
|
||||
|
||||
1. **crond is running** — if `inactive`, cron.daily never fires and logwatch never runs. No errors anywhere.
|
||||
2. **Postfix is installed and relaying** — logwatch depends on a working local MTA.
|
||||
3. **CA bundle exists (Fedora)** — missing `/etc/pki/tls/certs/ca-bundle.crt` breaks Postfix TLS relay. See [Fedora CA bundle fix](../../05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md).
|
||||
|
||||
## Diagnosing Silent Failures
|
||||
|
||||
```bash
|
||||
# Check if the defaults file was modified by a package upgrade
|
||||
rpm -V logwatch # Fedora
|
||||
dpkg -V logwatch # Debian
|
||||
|
||||
# Look for S.5....T. on the defaults file — means it was replaced
|
||||
# S = size, 5 = md5, T = timestamp changed
|
||||
```
|
||||
|
||||
## Fedora CA Bundle Missing — Postfix TLS Engine Unavailable
|
||||
|
||||
If the Fedora half of your fleet is silent but the Debian/Ubuntu half is fine, and your relayhost requires TLS, suspect a missing CA bundle. Symptom on the sending host:
|
||||
|
||||
```
|
||||
postfix/error: status=deferred (delivery temporarily suspended:
|
||||
TLS is required, but our TLS engine is unavailable)
|
||||
```
|
||||
|
||||
The tell that this is the CA bundle and not a postfix-internal problem: **dnf and curl are also broken on the box.** Run any `sudo dnf list` / `sudo curl https://...` and look for:
|
||||
|
||||
```
|
||||
Curl error (77): Problem with the SSL CA cert (path? access rights?)
|
||||
[error adding trust anchors from file: /etc/pki/tls/certs/ca-bundle.crt]
|
||||
```
|
||||
|
||||
That's the same path postfix's `smtp_tls_CAfile` defaults to. Every TLS client on the box is failing because a single symlink is missing.
|
||||
|
||||
### Diagnosis
|
||||
|
||||
```bash
|
||||
# Is the consumer-path symlink there?
|
||||
ls -la /etc/pki/tls/certs/ca-bundle.crt
|
||||
# Expected: lrwxrwxrwx ... -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
|
||||
|
||||
# Is the extracted bundle itself intact?
|
||||
ls -la /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
|
||||
sudo grep -c 'BEGIN CERTIFICATE' /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
|
||||
# Expected: ~140-150 certs, ~220 KB
|
||||
```
|
||||
|
||||
If the extracted bundle exists but the consumer-path symlink is gone, you've found it. `update-ca-trust extract` regenerates the `extracted/` paths but does **not** recreate the upstream-style symlink at `/etc/pki/tls/certs/ca-bundle.crt` — that symlink is shipped by the `ca-certificates` package and can be lost during a partial upgrade or a stray `rm`.
|
||||
|
||||
### Fix
|
||||
|
||||
```bash
|
||||
sudo ln -sfn /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
|
||||
/etc/pki/tls/certs/ca-bundle.crt
|
||||
sudo systemctl reload postfix
|
||||
sudo postqueue -f # drain deferred mail
|
||||
```
|
||||
|
||||
Verify with `sudo grep -c 'BEGIN CERTIFICATE' /etc/pki/tls/certs/ca-bundle.crt` (should match the extracted bundle's count) and `sudo dnf list --installed postfix` (should no longer show the curl error).
|
||||
|
||||
### Audit the rest of the Fedora fleet
|
||||
|
||||
Once you find one host with this issue, check the others — package events that broke one box may have broken its siblings:
|
||||
|
||||
```bash
|
||||
for host in $(your fleet | grep fedora); do
|
||||
echo "$host: $(ssh $host 'ls /etc/pki/tls/certs/ca-bundle.crt 2>&1' | tail -1)"
|
||||
done
|
||||
```
|
||||
|
||||
Hosts returning "No such file or directory" are silently broken. They won't fail loudly until something asks them to do TLS — which on a small homelab might be never until logwatch tries to mail you weeks later.
|
||||
|
||||
### Methodology note: postfix logs differ between distros
|
||||
|
||||
Don't trust a single log source when surveying a mixed fleet. **Fedora and majormail log postfix to journald** (`journalctl -u postfix`); **Debian/Ubuntu log to `/var/log/mail.log`** (and rotated `mail.log.1` / `mail.log.*.gz`). Querying journalctl on Ubuntu returns "no entries" even when mail is flowing — easy way to declare a working host broken. Always run `tail /var/log/mail.log` on Debian-family hosts and `journalctl -u postfix` on Fedora-family hosts.
|
||||
|
||||
## Bounce-source addresses must be real mailboxes
|
||||
|
||||
A subtle related class of bug: services like Watchtower, fail2ban, cron, and Netdata default to sending notifications **from** an identity that doesn't exist as a recipient — `watchtower@majorshouse.com`, `fail2ban@<host>.majorshouse.com`, `root@<host>.localdomain`. While the relayhost is healthy, nobody notices. The moment any delivery fails (network blip, recipient typo, queue overflow, the CA bundle bug above), the local MTA tries to bounce the original message back to that sender — finds no mailbox — and the bounce itself bounces. You get MAILER-DAEMON queue churn and `5.7.1 Relay access denied` rejections in your mail server logs.
|
||||
|
||||
Fix it once at the source: set `WATCHTOWER_NOTIFICATION_EMAIL_FROM`, fail2ban's `sender =`, and similar to a **real mailbox** on your mail server (e.g., `marcus@majorshouse.com`). Bounces then land somewhere a human can read them, and the noise disappears.
|
||||
|
||||
## Per-host config drift on cloud-image-derived servers
|
||||
|
||||
When fleet hosts are spun up from images (DigitalOcean droplet snapshots, Packer artifacts, cloud-init templates), three specific config drift patterns silently break notification mail. Each one looks fine in isolation; the combination produces "mail leaves the host with `250 OK queued` and disappears."
|
||||
|
||||
### 1. Packer/snapshot-leftover `myhostname` in postfix
|
||||
|
||||
A host built from a Packer-baked image often has `postfix myhostname = packer-<uuid>` baked into `main.cf` from the build process. The system hostname might have been correctly set by terraform/cloud-init at first boot, but postfix's `myhostname` was hardcoded during image build and was never overridden. Result: every outbound message-id and EHLO carries the Packer artifact name (e.g., `<20260509120011.7EB6ABD83C@packer-641079bc-bc17-b5e1-1425-be745d012d0b>`), no SPF/DKIM matches that name, and remote spam filters score it as suspicious.
|
||||
|
||||
**Detect:**
|
||||
|
||||
```bash
|
||||
postconf myhostname | grep -E 'packer-|builder-|<image-build-prefix>'
|
||||
```
|
||||
|
||||
**Fix:**
|
||||
|
||||
```bash
|
||||
hostnamectl set-hostname <real-fqdn>
|
||||
postconf -e 'myhostname = <real-fqdn>'
|
||||
sed -i '/^127\.0\.1\.1/d' /etc/hosts && \
|
||||
echo "127.0.1.1 <real-fqdn> <short-name>" >> /etc/hosts
|
||||
systemctl reload postfix
|
||||
```
|
||||
|
||||
### 2. Empty `relayhost` quietly forces public-MX delivery
|
||||
|
||||
If `postconf relayhost` returns an empty value, postfix doesn't fail — it just does an MX lookup for the destination domain and tries to deliver directly. For mail to your own mail server, that means going via the **public MX** (the domain's external MX record, e.g., `mail.majorshouse.com → 165.227.187.191:25`) instead of the **internal/Tailscale relay path** the rest of the fleet uses.
|
||||
|
||||
The public-MX path is subject to whatever spam filtering, content checks, and trust rules the receiving MX has configured for external traffic. Internal Tailscale-IP traffic typically gets a faster trust shortcut (e.g., bypass spamchk pipe). So this single configuration drift causes one host's mail to land in a different code path than its siblings — and then silently get filtered.
|
||||
|
||||
**Detect:** look for fleet hosts where `postconf relayhost` returns blank and compare to known-good siblings.
|
||||
|
||||
**Fix:** set `relayhost = [<mailserver-tailscale-ip>]:587` (or whatever port your fleet convention uses).
|
||||
|
||||
### 3. Stale SASL passwd map referencing a missing file
|
||||
|
||||
Postfix configurations migrated from a previous setup often retain `smtp_sasl_auth_enable = yes` and `smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd` even when no SASL is needed for the current relay path. If the actual `sasl_passwd` file isn't there (because the migration didn't carry it, or the new relay doesn't require auth), every send attempt produces:
|
||||
|
||||
```
|
||||
error: open database /etc/postfix/sasl_passwd.db: No such file or directory
|
||||
warning: smtp_sasl_password_maps lookup error
|
||||
status=deferred (local data error while talking to <relay>)
|
||||
```
|
||||
|
||||
Especially common after migrating from external SMTP (SendGrid, Mailgun, etc., which use SASL) to an internal Tailscale relay (which doesn't).
|
||||
|
||||
**Detect:**
|
||||
|
||||
```bash
|
||||
postconf -n | grep -E 'smtp_sasl_(auth_enable|password_maps)'
|
||||
[ -f /etc/postfix/sasl_passwd ] || echo "sasl_passwd file missing"
|
||||
```
|
||||
|
||||
**Fix — disable SASL if the new relay doesn't need it:**
|
||||
|
||||
```bash
|
||||
postconf -e 'smtp_sasl_auth_enable = no'
|
||||
postconf -e 'smtp_tls_wrappermode = no' # if switching from port 465 to 587
|
||||
postconf -X 'smtp_sasl_password_maps'
|
||||
systemctl reload postfix
|
||||
```
|
||||
|
||||
### Audit shortcut
|
||||
|
||||
For a quick per-host comparison across the fleet:
|
||||
|
||||
```bash
|
||||
for host in your fleet hosts; do
|
||||
echo "=== $host ==="
|
||||
ssh "$host" 'postconf myhostname relayhost smtp_sasl_auth_enable 2>&1' | head -3
|
||||
done
|
||||
```
|
||||
|
||||
Anomalies (Packer hostnames, blank relayhost, SASL enabled where siblings have it disabled) jump out immediately.
|
||||
|
||||
## Lesson Learned
|
||||
|
||||
Never customize `/usr/share/logwatch/default.conf/logwatch.conf`. Always use `/etc/logwatch/conf/logwatch.conf`. This applies to any software that has a "defaults" file and an "override" file — the override survives upgrades, the defaults file does not.
|
||||
|
||||
A second, broader lesson from the 2026-05-10 fleet outage: **silent fleet-wide email gaps are usually a stack of unrelated failures, not one cause.** That morning's investigation surfaced a missing CA bundle on two Fedora hosts, a postfix relayhost using a name that postfix's resolver couldn't handle, two services with non-mailbox sender addresses generating bounce churn, and a corrupt syslog-vs-journald assumption that hid working hosts. Each was minor in isolation. Together they made all seven hosts look broken when in fact only two were. Triage by ground-truth (what arrived in the destination mailbox) before assuming what's broken at the source.
|
||||
116
05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md
Normal file
116
05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
---
|
||||
title: "Fedora CA Bundle Missing Symlink — TLS Breaks Fleet-Wide"
|
||||
description: Hetzner-provisioned Fedora images may be missing the /etc/pki/tls/certs/ca-bundle.crt symlink, silently breaking Postfix TLS relay, curl, and dnf
|
||||
tags:
|
||||
- fedora
|
||||
- tls
|
||||
- postfix
|
||||
- ca-certificates
|
||||
- hetzner
|
||||
- troubleshooting
|
||||
status: published
|
||||
created: 2026-05-11
|
||||
updated: 2026-05-11
|
||||
---
|
||||
|
||||
# Fedora CA Bundle Missing Symlink
|
||||
|
||||
On Fedora, many TLS clients (Postfix, curl, dnf) look for the CA bundle at `/etc/pki/tls/certs/ca-bundle.crt`. This path is normally a symlink to `/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem`, shipped by the `ca-certificates` package.
|
||||
|
||||
On Hetzner Cloud Fedora images (observed on Fedora 44, May 2026), this symlink can be missing despite `ca-certificates` being installed. The extracted bundle exists, but the consumer-facing symlink does not.
|
||||
|
||||
## Symptoms
|
||||
|
||||
Postfix relay to a TLS-required upstream fails:
|
||||
|
||||
```
|
||||
postfix/smtp: cannot load Certification Authority data,
|
||||
CAfile="/etc/pki/tls/certs/ca-bundle.crt",
|
||||
CApath="/etc/pki/tls/certs": disabling TLS support
|
||||
```
|
||||
|
||||
If your relay requires TLS (port 465 with `smtp_tls_wrappermode = yes`, or `smtp_tls_security_level = encrypt`), mail silently queues as deferred. No bounce, no alert — just silence.
|
||||
|
||||
Other symptoms on the same box:
|
||||
|
||||
```bash
|
||||
# curl fails
|
||||
curl https://example.com
|
||||
# error: Problem with the SSL CA cert (path? access rights?)
|
||||
|
||||
# dnf fails
|
||||
dnf list --installed
|
||||
# Curl error (77): Problem with the SSL CA cert
|
||||
```
|
||||
|
||||
## Diagnosis
|
||||
|
||||
```bash
|
||||
# Check the symlink
|
||||
ls -la /etc/pki/tls/certs/ca-bundle.crt
|
||||
# Expected: symlink -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
|
||||
# Broken: "No such file or directory"
|
||||
|
||||
# Verify the extracted bundle exists
|
||||
ls -la /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
|
||||
# Should exist (~220 KB, ~140-150 certs)
|
||||
|
||||
# Confirm the package is installed
|
||||
rpm -q ca-certificates
|
||||
# Should return a version string
|
||||
```
|
||||
|
||||
If the extracted bundle exists but the symlink at `/etc/pki/tls/certs/ca-bundle.crt` is missing, that's the problem.
|
||||
|
||||
## Fix
|
||||
|
||||
```bash
|
||||
sudo ln -sf /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem \
|
||||
/etc/pki/tls/certs/ca-bundle.crt
|
||||
sudo systemctl restart postfix
|
||||
sudo postqueue -f # flush any deferred mail
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```bash
|
||||
# Symlink exists
|
||||
ls -la /etc/pki/tls/certs/ca-bundle.crt
|
||||
|
||||
# Postfix can relay
|
||||
echo "Subject: TLS test" | sendmail -v marcus@majorshouse.com
|
||||
|
||||
# curl works
|
||||
curl -sI https://example.com | head -1
|
||||
```
|
||||
|
||||
## Fleet Audit
|
||||
|
||||
If one Hetzner-provisioned Fedora host has this issue, check the others:
|
||||
|
||||
```bash
|
||||
for host in majordiscord majorlab majorhome majormail; do
|
||||
echo "$host: $(ssh root@$host 'ls /etc/pki/tls/certs/ca-bundle.crt 2>&1' | tail -1)"
|
||||
done
|
||||
```
|
||||
|
||||
Hosts returning "No such file or directory" are silently broken for all TLS operations.
|
||||
|
||||
## Why This Happens
|
||||
|
||||
`update-ca-trust extract` regenerates the files under `/etc/pki/ca-trust/extracted/` but does not create the legacy consumer-path symlink at `/etc/pki/tls/certs/ca-bundle.crt`. That symlink is shipped by the `ca-certificates` RPM. On cloud images built from minimal installs or snapshot-based provisioning, the symlink can be lost during image creation or a partial upgrade.
|
||||
|
||||
## Prevention
|
||||
|
||||
Add to your provisioning checklist (see [VPS Migration Baseline Checklist](../../02-selfhosting/cloud/vps-migration-baseline-checklist.md)):
|
||||
|
||||
```bash
|
||||
# Fedora provisioning — verify CA bundle symlink
|
||||
ls /etc/pki/tls/certs/ca-bundle.crt || \
|
||||
ln -sf /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/tls/certs/ca-bundle.crt
|
||||
```
|
||||
|
||||
## Related
|
||||
|
||||
- [Logwatch Fleet Setup](../../02-selfhosting/monitoring/logwatch-fleet-setup.md) — logwatch depends on a working Postfix relay, which depends on TLS, which depends on this symlink
|
||||
- [VPS Migration Baseline Checklist](../../02-selfhosting/cloud/vps-migration-baseline-checklist.md) — includes CA bundle verification step
|
||||
14
SUMMARY.md
14
SUMMARY.md
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
created: 2026-04-02T16:03
|
||||
updated: 2026-05-02T17:16
|
||||
updated: 2026-05-11T07:35
|
||||
---
|
||||
* [Home](index.md)
|
||||
* [Linux & Sysadmin](01-linux/index.md)
|
||||
|
|
@ -28,14 +28,17 @@ updated: 2026-05-02T17:16
|
|||
* [Wake-on-LAN via Router SSH](02-selfhosting/dns-networking/wake-on-lan-router-ssh.md)
|
||||
* [Pi-hole v6 Group Management — Per-Client DNS Rules](02-selfhosting/dns-networking/pihole-v6-group-management.md)
|
||||
* [AWS S3 Cost Management](02-selfhosting/cloud/aws-s3-cost-management.md)
|
||||
* [VPS Migration Baseline Checklist](02-selfhosting/cloud/vps-migration-baseline-checklist.md)
|
||||
* [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md)
|
||||
* [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md)
|
||||
* [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md)
|
||||
* [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md)
|
||||
* [Netdata SELinux AVC Denial Monitoring](02-selfhosting/monitoring/netdata-selinux-avc-chart.md)
|
||||
* [Netdata n8n Enriched Alert Emails](02-selfhosting/monitoring/netdata-n8n-enriched-alerts.md)
|
||||
* [Logwatch Fleet Setup — Surviving Package Upgrades](02-selfhosting/monitoring/logwatch-fleet-setup.md)
|
||||
* [Updating n8n Running in Docker](02-selfhosting/services/updating-n8n-docker.md)
|
||||
* [Mastodon Instance Tuning](02-selfhosting/services/mastodon-instance-tuning.md)
|
||||
* [Mastodon — The `--prune-profiles` Trap and How to Recover](02-selfhosting/services/mastodon-prune-profiles-trap.md)
|
||||
* [Ghost Email Configuration with Mailgun](02-selfhosting/services/ghost-smtp-mailgun-setup.md)
|
||||
* [Claude Code Remote Control — Mobile Access to a Persistent Host Session](02-selfhosting/services/claude-code-remote-control.md)
|
||||
* [Linux Server Hardening Checklist](02-selfhosting/security/linux-server-hardening-checklist.md)
|
||||
|
|
@ -52,6 +55,7 @@ updated: 2026-05-02T17:16
|
|||
* [SSH Hardening Fleet-Wide with Ansible](02-selfhosting/security/ssh-hardening-ansible-fleet.md)
|
||||
* [ClamAV Fleet Deployment with Ansible](02-selfhosting/security/clamav-fleet-deployment.md)
|
||||
* [Fail2Ban Digest Mode — Fleet-Wide Quiet Alerts](02-selfhosting/security/fail2ban-digest-mode-fleet.md)
|
||||
* [Apache CVE-2026-23918 — HTTP/2 Double Free Mitigation](02-selfhosting/security/apache-cve-2026-23918-http2-mitigation.md)
|
||||
* [Open Source & Alternatives](03-opensource/index.md)
|
||||
* [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md)
|
||||
* [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md)
|
||||
|
|
@ -73,6 +77,9 @@ updated: 2026-05-02T17:16
|
|||
* [Tailscale SSH: Unexpected Re-Authentication Prompt](05-troubleshooting/networking/tailscale-ssh-reauth-prompt.md)
|
||||
* [Fail2ban & UFW Rule Bloat Cleanup](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md)
|
||||
* [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md)
|
||||
* [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md)
|
||||
* [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md)
|
||||
* [Castopod Posts Don't Appear on Mastodon — Diagnosing the Federation Path](05-troubleshooting/security/castopod-broadcast-not-on-mastodon.md)
|
||||
* [Nextcloud AIO Unhealthy 20h After Nightly Update](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md)
|
||||
* [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md)
|
||||
* [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md)
|
||||
|
|
@ -90,15 +97,20 @@ updated: 2026-05-02T17:16
|
|||
* [Windows OpenSSH Server (sshd) Stops After Reboot](05-troubleshooting/networking/windows-sshd-stops-after-reboot.md)
|
||||
* [Windows OpenSSH: WSL Default Shell Breaks Remote Commands](05-troubleshooting/networking/windows-openssh-wsl-default-shell-breaks-remote-commands.md)
|
||||
* [Pi-hole AI Blocklist Blocks Claude Desktop (ERR_CONNECTION_REFUSED)](05-troubleshooting/networking/pihole-blocks-claude-desktop.md)
|
||||
* [Claude Desktop MCP Server Started via wsl.exe Sees Empty Environment (WSLENV)](05-troubleshooting/wsl-env-claude-desktop-mcp.md)
|
||||
* [Claude Desktop MCP Mass-Disconnect After Blocking SSH Reboot](05-troubleshooting/claude-desktop-mcp-mass-disconnect-blocking-reboot.md)
|
||||
* [Patching PHP 8.4 Implicit-Nullable Deprecations in Vendor Packages](05-troubleshooting/php-84-vendor-implicit-nullable-patch.md)
|
||||
* [Ollama Drops Off Tailscale When Mac Sleeps](05-troubleshooting/ollama-macos-sleep-tailscale-disconnect.md)
|
||||
* [Ollama: `ollama run` with Piped Stdin Bypasses Chat Template + SYSTEM Prompt](05-troubleshooting/ollama-chat-template-pipe-stdin-bypass.md)
|
||||
* [rsync over Tailscale: Hung in TCP Teardown After Transfer Completes](05-troubleshooting/networking/rsync-tailscale-teardown-stall.md)
|
||||
* [iOS Tailscale Clients Report HostName="localhost" — Breaks /etc/hosts Generators](05-troubleshooting/networking/tailscale-status-json-hostname-localhost-ios.md)
|
||||
* [macOS: Repeating Alert Tone from Mirrored iPhone Notification](05-troubleshooting/macos-mirrored-notification-alert-loop.md)
|
||||
* [ClamAV CPU Spike: Safe Scheduling with nice/ionice](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md)
|
||||
* [Fedora CA Bundle Missing Symlink — TLS Breaks Fleet-Wide](05-troubleshooting/security/fedora-ca-bundle-missing-symlink.md)
|
||||
* [Ansible: Vault Password File Not Found](05-troubleshooting/ansible-vault-password-file-missing.md)
|
||||
* [Ansible: ansible.cfg Ignored on WSL2 Windows Mounts](05-troubleshooting/ansible-wsl2-world-writable-mount-ignores-cfg.md)
|
||||
* [Ansible: SSH Timeout During dnf upgrade on Fedora Hosts](05-troubleshooting/ansible-ssh-timeout-dnf-upgrade.md)
|
||||
* [Ansible: regex_search Capture-Group Argument Fails in set_fact](05-troubleshooting/ansible-regex-search-set-fact-capture-group.md)
|
||||
* [Fedora Networking & Kernel Troubleshooting](05-troubleshooting/fedora-networking-kernel-recovery.md)
|
||||
* [Systemd Session Scope Fails at Login](05-troubleshooting/systemd/session-scope-failure-at-login.md)
|
||||
* [wget/curl: URLs with Special Characters Fail in Bash](05-troubleshooting/wget-url-special-characters.md)
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue