diff --git a/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md b/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md new file mode 100644 index 0000000..a54a7a6 --- /dev/null +++ b/05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md @@ -0,0 +1,135 @@ +# Docker & Caddy Recovery After Reboot (Fedora + SELinux) + +## 🛑 Problem + +After a system reboot on **majorlab** (Fedora 43, SELinux Enforcing), Docker containers and all Caddy-proxied services become unreachable. Browsers may show connection errors or 502 Bad Gateway responses. + +## 🔍 Diagnosis + +Three separate failures occur in sequence: + +### 1. Docker fails to start + +```bash +systemctl status docker.service +# → Active: inactive (dead) +# → Dependency failed for docker.service + +systemctl status docker.socket +# → Active: failed (Result: resources) +# → Failed to create listening socket (/run/docker.sock): Invalid argument +``` + +**Cause:** `docker.socket` is disabled, so Docker's socket activation fails and `docker.service` never starts. All containers are down. + +--- + +### 2. Caddy fails to bind ports + +```bash +journalctl -u caddy -n 20 +# → Error: listen tcp :4443: bind: permission denied +# → Error: listen tcp :8448: bind: permission denied +``` + +**Cause:** SELinux's `http_port_t` type does not include ports `4443` (Tailscale HTTPS) or `8448` (Matrix federation), so Caddy is denied when trying to bind them. + +--- + +### 3. Caddy returns 502 Bad Gateway + +Even after Caddy starts, all reverse proxied services return 502. + +```bash +journalctl -u caddy | grep "permission denied" +# → dial tcp 127.0.0.1:: connect: permission denied +``` + +**Cause:** The SELinux boolean `httpd_can_network_connect` is off, preventing Caddy from making outbound connections to upstream services. + +--- + +## ✅ Solution + +### Step 1 — Re-enable and start Docker + +```bash +sudo systemctl enable docker.socket +sudo systemctl start docker.socket +sudo systemctl start docker.service +``` + +Verify containers are up: + +```bash +sudo docker ps -a +``` + +--- + +### Step 2 — Add missing ports to SELinux http_port_t + +```bash +sudo semanage port -m -t http_port_t -p tcp 4443 +sudo semanage port -a -t http_port_t -p tcp 8448 +``` + +Verify: + +```bash +sudo semanage port -l | grep http_port_t +# Should include 4443 and 8448 +``` + +--- + +### Step 3 — Enable httpd_can_network_connect + +```bash +sudo setsebool -P httpd_can_network_connect on +``` + +The `-P` flag makes this persistent across reboots. + +--- + +### Step 4 — Start Caddy + +```bash +sudo systemctl restart caddy +systemctl is-active caddy +# → active +``` + +--- + +## 🔁 Why This Happens + +| Issue | Root Cause | +|---|---| +| Docker down | `docker.socket` was disabled (not just stopped) — survives reboots until explicitly enabled | +| Port bind denied | SELinux requires non-standard ports to be explicitly added to `http_port_t` — this is not automatic on upgrades or reinstalls | +| 502 on all proxied services | `httpd_can_network_connect` defaults to `off` on Fedora — must be set once per installation | + +--- + +## 🔎 Quick Diagnostic Commands + +```bash +# Check Docker +systemctl status docker.socket docker.service +sudo docker ps -a + +# Check Caddy +systemctl status caddy +journalctl -u caddy -n 30 + +# Check SELinux booleans +getsebool httpd_can_network_connect + +# Check allowed HTTP ports +sudo semanage port -l | grep http_port_t + +# Test upstream directly (bypass Caddy) +curl -sv http://localhost:8086 +``` diff --git a/05-troubleshooting/index.md b/05-troubleshooting/index.md index 3d44700..9bfc177 100644 --- a/05-troubleshooting/index.md +++ b/05-troubleshooting/index.md @@ -6,3 +6,4 @@ Practical fixes for common Linux, networking, and application problems. - [Obsidian Cache Hang Recovery](obsidian-cache-hang-recovery.md) - [yt-dlp Fedora JS Challenge](yt-dlp-fedora-js-challenge.md) - [MajorWiki Setup & Publishing Pipeline](majwiki-setup-and-pipeline.md) +- [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](docker-caddy-selinux-post-reboot-recovery.md) diff --git a/SUMMARY.md b/SUMMARY.md index 936b123..6d73af9 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,19 +1,10 @@ -* [Home](index.md) -* [Linux & Sysadmin](01-linux/index.md) - * [Files & Permissions](01-linux/files-permissions/) - * [Process Management](01-linux/process-management/) - * [Networking](01-linux/networking/) - * [Package Management](01-linux/packages/) - * [Shell & Scripting](01-linux/shell-scripting/) - * [Distro-Specific](01-linux/distro-specific/) - * [Self-Hosting](02-selfhosting/index.md) - * [Docker & Containers](02-selfhosting/docker/) - * [Reverse Proxies](02-selfhosting/reverse-proxy/) - * [DNS & Networking](02-selfhosting/dns-networking/) - * [Storage & Backup](02-selfhosting/storage-backup/) - * [Monitoring](02-selfhosting/monitoring/) - * [Security](02-selfhosting/security/) - * [Streaming](04-streaming/index.md) - * [OBS Studio](04-streaming/obs/) - * [Troubleshooting](05-troubleshooting/index.md) - * 05-troubleshooting/*.md \ No newline at end of file +* [Home](index.md) +* [Linux & Sysadmin](01-linux/index.md) + * [Introduction](01-linux/index.md) +* [Self-Hosting](02-selfhosting/index.md) + * [Introduction](02-selfhosting/index.md) +* [Streaming](04-streaming/index.md) + * [Introduction](04-streaming/index.md) +* [Troubleshooting](05-troubleshooting/index.md) + * [ISP SNI Filtering & Caddy](05-troubleshooting/isp-sni-filtering-caddy.md) + * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md)