chore: link vault wiki to Gitea

02-selfhosting/docker/debugging-broken-docker-containers.md

@@ -0,0 +1,168 @@
+---
+title: "Debugging Broken Docker Containers"
+domain: selfhosting
+category: docker
+tags: [docker, troubleshooting, debugging, containers, logs]
+status: published
+created: 2026-03-08
+updated: 2026-03-08
+---
+
+# Debugging Broken Docker Containers
+
+When something in Docker breaks, there's a sequence to work through. Check logs first, inspect the container second, deal with network and permissions third. Resist the urge to rebuild immediately — most failures tell you what's wrong if you look.
+
+## The Short Answer
+
+```bash
+# Step 1: check logs
+docker logs containername
+
+# Step 2: inspect the container
+docker inspect containername
+
+# Step 3: get a shell inside
+docker exec -it containername /bin/bash
+# or /bin/sh if bash isn't available
+```
+
+## The Debugging Sequence
+
+### 1. Check the logs first
+
+Before anything else:
+
+```bash
+docker logs containername
+
+# Follow live output
+docker logs -f containername
+
+# Last 100 lines
+docker logs --tail 100 containername
+
+# With timestamps
+docker logs --timestamps containername
+```
+
+Most failures announce themselves in the logs. Crash loops, config errors, missing environment variables — it's usually right there.
+
+### 2. Check if the container is actually running
+
+```bash
+docker ps
+
+# Show stopped containers too
+docker ps -a
+```
+
+If the container shows as `Exited (1)` or any non-zero exit code, it crashed. The logs will usually tell you why.
+
+### 3. Inspect the container
+
+`docker inspect` dumps everything — environment variables, mounts, network config, the actual command being run:
+
+```bash
+docker inspect containername
+```
+
+Too verbose? Filter for the part you need:
+
+```bash
+# Just the mounts
+docker inspect --format='{{json .Mounts}}' containername | jq
+
+# Just the environment variables
+docker inspect --format='{{json .Config.Env}}' containername | jq
+
+# Exit code
+docker inspect --format='{{.State.ExitCode}}' containername
+```
+
+### 4. Get a shell inside
+
+If the container is running but misbehaving:
+
+```bash
+docker exec -it containername /bin/bash
+```
+
+If it crashed and won't stay up, override the entrypoint to get in anyway:
+
+```bash
+docker run -it --entrypoint /bin/bash imagename:tag
+```
+
+### 5. Check port conflicts
+
+Container starts but the service isn't reachable:
+
+```bash
+# See what ports are mapped
+docker ps --format "table {{.Names}}\t{{.Ports}}"
+
+# See what's using a port on the host
+sudo ss -tlnp | grep :8080
+```
+
+Two containers can't bind the same host port. If something else grabbed the port first, the container will start but the port won't be accessible.
+
+### 6. Check volume permissions
+
+Permission errors inside containers are almost always a UID mismatch. The user inside the container doesn't own the files on the host volume.
+
+```bash
+# Check ownership of the mounted directory on the host
+ls -la /path/to/host/volume
+
+# Find out what UID the container runs as
+docker inspect --format='{{.Config.User}}' containername
+```
+
+Fix by chowning the host directory to match, or by explicitly setting the user in your compose file:
+
+```yaml
+services:
+  myapp:
+    image: myapp:latest
+    user: "1000:1000"
+    volumes:
+      - ./data:/app/data
+```
+
+### 7. Recreate cleanly when needed
+
+If you've made config changes and things are in a weird state:
+
+```bash
+docker compose down
+docker compose up -d
+
+# Nuclear option — also removes volumes (data gone)
+docker compose down -v
+```
+
+Don't jump straight to the nuclear option. Only use `-v` if you want a completely clean slate and don't care about the data.
+
+## Common Failure Patterns
+
+| Symptom | Likely cause | Where to look |
+|---|---|---|
+| Exits immediately | Config error, missing env var | `docker logs` |
+| Keeps restarting | Crash loop — app failing to start | `docker logs`, exit code |
+| Port not reachable | Port conflict or wrong binding | `docker ps`, `ss -tlnp` |
+| Permission denied inside container | UID mismatch on volume | `ls -la` on host path |
+| "No such file or directory" | Wrong mount path or missing file | `docker inspect` mounts |
+| Container runs but service is broken | App config error, not Docker | shell in, check app logs |
+
+## Gotchas & Notes
+
+- **`docker restart` doesn't pick up compose file changes.** Use `docker compose up -d` to apply changes.
+- **Logs persist after a container exits.** You can still `docker logs` a stopped container — useful for post-mortem on crashes.
+- **If a container won't stay up long enough to exec into it**, use the entrypoint override (`--entrypoint /bin/bash`) with `docker run` against the image directly.
+- **Watch out for cached layers on rebuild.** If you're rebuilding an image and the behavior doesn't change, add `--no-cache` to `docker build`.
+
+## See Also
+
+- [[docker-vs-vms-homelab]]
+- [[tuning-netdata-web-log-alerts]]