chore: link vault wiki to Gitea

02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md

@@ -0,0 +1,88 @@
+---
+title: Tuning Netdata Web Log Alerts
+domain: selfhosting
+category: monitoring
+tags:
+  - netdata
+  - apache
+  - monitoring
+  - alerts
+  - ubuntu
+status: published
+created: '2026-03-06'
+updated: '2026-03-08'
+---
+
+# Tuning Netdata Web Log Alerts
+
+To stop Netdata's `web_log_1m_redirects` alert from firing on normal HTTP-to-HTTPS redirect traffic, edit `/etc/netdata/health.d/web_log.conf` and raise the redirect threshold to 80%, then reload with `netdatacli reload-health`. The default threshold is too sensitive for any server that forces HTTPS — automated traffic hits port 80, gets a 301, and Netdata flags it as a WARNING even though nothing is wrong.
+
+## The Short Answer
+
+```bash
+sudo /etc/netdata/edit-config health.d/web_log.conf
+```
+
+Change the `warn` line in the `web_log_1m_redirects` template to:
+
+```bash
+warn: ($web_log_1m_requests > 120) ? ($this > (($status >= $WARNING ) ? ( 1 ) : ( 80 )) ) : ( 0 )
+```
+
+Then reload:
+
+```bash
+netdatacli reload-health
+```
+
+## Background
+
+Production nodes forcing HTTPS see a lot of 301s. The default Netdata threshold is too sensitive for sites with a high bot-to-human ratio — it was designed for environments where redirects are unexpected, not standard operating procedure.
+
+The tuned logic warns only when there are more than 120 requests/minute AND redirects exceed 80% of traffic (dropping to 1% once already in WARNING state to prevent flapping).
+
+## Steps
+
+1. Identify what's actually generating the redirects
+
+```bash
+# Check status code distribution
+awk '{print $9}' /var/log/apache2/access.log | sort | uniq -c | sort -nr
+
+# Top redirected URLs
+awk '$9 == "301" {print $7}' /var/log/apache2/access.log | sort | uniq -c | sort -nr | head -n 10
+```
+
+2. Open the Netdata health config
+
+```bash
+sudo /etc/netdata/edit-config health.d/web_log.conf
+```
+
+3. Find the `web_log_1m_redirects` template and update the `warn` line
+
+```bash
+warn: ($web_log_1m_requests > 120) ? ($this > (($status >= $WARNING ) ? ( 1 ) : ( 80 )) ) : ( 0 )
+```
+
+4. Reload without restarting the service
+
+```bash
+netdatacli reload-health
+```
+
+5. Verify the alert cleared
+
+```bash
+curl -s http://localhost:19999/api/v1/alarms?all | grep -A 15 "web_log_1m_redirec"
+```
+
+## Gotchas & Notes
+
+- **Ubuntu/Debian only:** The `edit-config` path and `apache2` log location are Debian-specific. On Fedora/RHEL the log is at `/var/log/httpd/access_log`.
+- **The 1% recovery threshold is intentional:** Without it, the alert will flap between WARNING and CLEAR constantly on busy sites. The hysteresis keeps it stable once triggered.
+- **Adjust the 120 req/min floor to your traffic:** Low-traffic sites may need a lower threshold; high-traffic sites may need higher.
+
+## See Also
+
+- [[Netdata service monitoring]]