diff --git a/05-troubleshooting/gitea-runner-boot-race-network-target.md b/05-troubleshooting/gitea-runner-boot-race-network-target.md new file mode 100644 index 0000000..4d64c4b --- /dev/null +++ b/05-troubleshooting/gitea-runner-boot-race-network-target.md @@ -0,0 +1,84 @@ +# Gitea Actions Runner: Boot Race Condition Fix + +If your `gitea-runner` (act_runner) service fails to start on boot — crash-looping and eventually hitting systemd's restart rate limit — the service is likely starting before DNS is available. + +## Symptoms + +- `gitea-runner.service` enters a crash loop on boot +- `journalctl -u gitea-runner` shows connection/DNS errors on startup: + ``` + dial tcp: lookup git.example.com: no such host + ``` + or similar resolution failures +- Service eventually stops retrying (systemd restart rate limit reached) +- `systemctl status gitea-runner` shows `(Result: start-limit-hit)` after reboot +- Service works fine if started manually after boot completes + +## Why It Happens + +`After=network.target` only guarantees that the network **interfaces are configured** — not that DNS resolution is functional. systemd-resolved (or your local resolver) starts slightly later. `act_runner` tries to connect to the Gitea instance by hostname on startup, the DNS lookup fails, and the process exits. + +With the default `Restart=always` and no `RestartSec`, systemd restarts the service immediately. After 5 rapid failures within the default burst window (10 attempts in 2 minutes), systemd hits the rate limit and stops restarting. + +## Fix + +### 1. Update the Service File + +Edit `/etc/systemd/system/gitea-runner.service`: + +```ini +[Unit] +Description=Gitea Actions Runner +After=network-online.target +Wants=network-online.target + +[Service] +User=deploy +WorkingDirectory=/opt/gitea-runner +ExecStart=/opt/gitea-runner/act_runner daemon +Restart=always +RestartSec=10 + +[Install] +WantedBy=multi-user.target +``` + +Key changes: +- `After=network-online.target` + `Wants=network-online.target` — waits for full network stack including DNS +- `RestartSec=10` — adds a 10-second delay between restart attempts, preventing rapid failure bursts from hitting the rate limit + +### 2. Add a Local /etc/hosts Entry (Optional but Recommended) + +If your Gitea instance is on the same local network or reachable via Tailscale, add an entry to `/etc/hosts` so act_runner can resolve it without depending on external DNS: + +``` +127.0.0.1 git.example.com +``` + +Replace `git.example.com` with your Gitea hostname and the IP with the correct local address. This makes resolution instantaneous and eliminates the DNS dependency entirely for startup. + +### 3. Reload and Restart + +```bash +sudo systemctl daemon-reload +sudo systemctl restart gitea-runner +sudo systemctl status gitea-runner +``` + +Verify it shows `active (running)` and stays that way. Then reboot and confirm it comes up automatically. + +## Why `network-online.target` and Not `network.target` + +| Target | What it guarantees | +|---|---| +| `network.target` | Network interfaces are configured (IP assigned) | +| `network-online.target` | Network is fully operational (DNS resolvers reachable) | + +Services that need to make outbound network connections (especially DNS lookups) on startup should always use `network-online.target`. This includes: mail servers, monitoring agents, CI runners, anything that connects to an external host by name. + +> [!note] `network-online.target` can add a few seconds to boot time since systemd waits for the network stack to fully initialize. For server contexts this is always the right tradeoff. + +## Related + +- [Managing Linux Services with systemd](../01-linux/process-management/managing-linux-services-systemd-ansible.md) +- [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index e6234dc..1fffcea 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -13,8 +13,12 @@ Practical fixes for common Linux, networking, and application problems. ## 📦 Docker & Systems - [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) +- [Gitea Actions Runner: Boot Race Condition Fix](gitea-runner-boot-race-network-target.md) - [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) +## 🔒 SELinux +- [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](selinux-dovecot-vmail-context.md) + ## 📝 Application Specific - [Obsidian Vault Recovery — Loading Cache Hang](obsidian-cache-hang-recovery.md) - [Gemini CLI Manual Update](gemini-cli-manual-update.md) diff --git a/05-troubleshooting/selinux-dovecot-vmail-context.md b/05-troubleshooting/selinux-dovecot-vmail-context.md new file mode 100644 index 0000000..2ca06cf --- /dev/null +++ b/05-troubleshooting/selinux-dovecot-vmail-context.md @@ -0,0 +1,103 @@ +# SELinux: Fixing Dovecot Mail Spool Context (/var/vmail) + +If Dovecot is generating SELinux AVC denials and mail delivery or retrieval is broken on a Fedora/RHEL system with SELinux enforcing, the `/var/vmail` directory tree likely has incorrect file contexts. + +## Symptoms + +- Thousands of AVC denials in `/var/log/audit/audit.log` for Dovecot processes +- Denials reference `var_t` context on files under `/var/vmail/` +- Mail delivery may fail silently; IMAP folders may appear empty or inaccessible +- `ausearch -m avc -ts recent` shows denials like: + ``` + type=AVC msg=audit(...): avc: denied { write } for pid=... comm="dovecot" name="..." scontext=system_u:system_r:dovecot_t:s0 tcontext=system_u:object_r:var_t:s0 + ``` + +## Why It Happens + +SELinux requires files to have the correct security context for the process that accesses them. When Postfix/Dovecot are installed on a fresh system and `/var/vmail` is created manually (or by the mail stack installer), the directory may inherit the default `var_t` context from `/var/` rather than the mail-specific `mail_spool_t` context Dovecot expects. + +The correct context for the entire `/var/vmail` tree is `mail_spool_t` — including the `tmp/` subdirectories inside each Maildir folder. + +> [!warning] Do NOT apply `dovecot_tmp_t` to Maildir `tmp/` directories +> `dovecot_tmp_t` is for Dovecot's own process-level temp files, not for Maildir `tmp/` folders. Postfix's virtual delivery agent writes to `tmp/` when delivering new mail. Applying `dovecot_tmp_t` will block Postfix from delivering any mail, silently deferring all messages with `Permission denied`. + +## Fix + +### 1. Check Current Context + +```bash +ls -Zd /var/vmail/ +ls -Z /var/vmail/example.com/user/ +ls -Zd /var/vmail/example.com/user/tmp/ +``` + +If you see `var_t` instead of `mail_spool_t`, the contexts need to be set. If you see `dovecot_tmp_t` on `tmp/`, that needs to be corrected too. + +### 2. Define the Correct File Context Rule + +One rule covers everything — including `tmp/`: + +```bash +sudo semanage fcontext -a -t mail_spool_t "/var/vmail(/.*)?" +``` + +If you previously added a `dovecot_tmp_t` rule for `tmp/` directories, remove it: + +```bash +# Check for an erroneous dovecot_tmp_t rule +sudo semanage fcontext -l | grep vmail + +# If you see one like "/var/vmail(/.*)*/tmp(/.*)?" with dovecot_tmp_t, delete it: +sudo semanage fcontext -d "/var/vmail(/.*)*/tmp(/.*)?" +``` + +### 3. Apply the Labels + +```bash +sudo restorecon -Rv /var/vmail +``` + +This relabels all existing files. On a mail server with many users and messages, this may take a moment and will print every relabeled path. + +### 4. Verify + +```bash +ls -Zd /var/vmail/ +ls -Zd /var/vmail/example.com/user/tmp/ +``` + +Both should show `mail_spool_t`: +``` +system_u:object_r:mail_spool_t:s0 /var/vmail/ +system_u:object_r:mail_spool_t:s0 /var/vmail/example.com/user/tmp/ +``` + +### 5. Flush Deferred Mail + +If mail was queued while the context was wrong, flush it: + +```bash +postqueue -f +postqueue -p # should be empty shortly +``` + +### 6. Check That Denials Stopped + +```bash +ausearch -m avc -ts recent | grep dovecot +``` + +No output = no new denials. + +## Key Notes + +- **One rule is enough** — `"/var/vmail(/.*)?"` with `mail_spool_t` covers every file and directory under `/var/vmail`, including all `tmp/` subdirectories. +- **`semanage fcontext` is persistent** — the rules survive reboots and `restorecon` calls. You only need to run `semanage` once. +- **`restorecon` applies current rules to existing files** — run it after any `semanage` change and any time you manually create directories. +- **New mail directories are labeled automatically** — SELinux applies the registered `semanage` rules to any new files created under `/var/vmail`. +- **`var_t` context is the default for `/var/`** — any directory created under `/var/` without a specific `semanage` rule will inherit `var_t`. This is almost never correct for service data directories. + +## Related + +- [Linux Server Hardening Checklist](../02-selfhosting/security/linux-server-hardening-checklist.md) +- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) diff --git a/MajorWiki-Deploy-Status.md b/MajorWiki-Deploy-Status.md index 447b830..15a0318 100644 --- a/MajorWiki-Deploy-Status.md +++ b/MajorWiki-Deploy-Status.md @@ -31,7 +31,7 @@ DNS record and Caddy entry have been removed. ## Content -- 37 articles across 5 domains +- 39 articles across 5 domains - Source of truth: `MajorVault/20-Projects/MajorTwin/08-Wiki/` - Deployed via Gitea webhook (push from MajorAir → auto-pull on majorlab) diff --git a/README.md b/README.md index 59af5d5..d8affe2 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > **Last updated:** 2026-03-14 -**Article count:** 37 +**Article count:** 39 ## Domains @@ -13,7 +13,7 @@ | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | | 🔓 Open Source Tools | `03-opensource/` | 9 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 10 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 12 | --- @@ -109,6 +109,8 @@ - [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora - [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline +- [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) — fixing act_runner crash loop on boot caused by DNS not ready at startup +- [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) — fixing thousands of AVC denials when /var/vmail has wrong SELinux context --- @@ -116,6 +118,8 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) | Troubleshooting | +| 2026-03-14 | [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) | Troubleshooting | | 2026-03-14 | [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md) | Troubleshooting | | 2026-03-14 | [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) | Open Source | | 2026-03-14 | [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) | Open Source | diff --git a/SUMMARY.md b/SUMMARY.md index 281840b..5d2a5a7 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -40,3 +40,5 @@ * [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) * [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) * [MajorWiki Setup & Publishing Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) + * [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) + * [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) diff --git a/index.md b/index.md index 22f7019..b50118d 100644 --- a/index.md +++ b/index.md @@ -3,7 +3,7 @@ > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-03-14 -> **Article count:** 37 +> **Article count:** 39 ## Domains @@ -13,7 +13,7 @@ | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 8 | | 🔓 Open Source Tools | `03-opensource/` | 9 | | 🎙️ Streaming & Podcasting | `04-streaming/` | 1 | -| 🔧 General Troubleshooting | `05-troubleshooting/` | 10 | +| 🔧 General Troubleshooting | `05-troubleshooting/` | 12 | --- @@ -109,6 +109,8 @@ - [yt-dlp YouTube JS Challenge Fix on Fedora](05-troubleshooting/yt-dlp-fedora-js-challenge.md) — fixing YouTube JS challenge solver errors and missing formats on Fedora - [Gemini CLI Manual Update](05-troubleshooting/gemini-cli-manual-update.md) — how to manually update the Gemini CLI when automatic updates fail - [MajorWiki Setup & Pipeline](05-troubleshooting/majwiki-setup-and-pipeline.md) — setting up MajorWiki and the Obsidian → Gitea → MkDocs publishing pipeline +- [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) — fixing act_runner crash loop on boot caused by DNS not ready at startup +- [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) — fixing thousands of AVC denials when /var/vmail has wrong SELinux context --- @@ -116,6 +118,8 @@ | Date | Article | Domain | |---|---|---| +| 2026-03-14 | [SELinux: Fixing Dovecot Mail Spool Context (/var/vmail)](05-troubleshooting/selinux-dovecot-vmail-context.md) | Troubleshooting | +| 2026-03-14 | [Gitea Actions Runner: Boot Race Condition Fix](05-troubleshooting/gitea-runner-boot-race-network-target.md) | Troubleshooting | | 2026-03-14 | [Mail Client Stops Receiving: Fail2ban IMAP Self-Ban](05-troubleshooting/networking/fail2ban-imap-self-ban-mail-client.md) | Troubleshooting | | 2026-03-14 | [SearXNG: Private Self-Hosted Search](03-opensource/alternatives/searxng.md) | Open Source | | 2026-03-14 | [FreshRSS: Self-Hosted RSS Reader](03-opensource/alternatives/freshrss.md) | Open Source |