wiki: add SELinux AVC chart, enriched alerts, new server setup, and pending articles; update indexes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

02-selfhosting/monitoring/netdata-new-server-setup.md

@@ -0,0 +1,161 @@
+---
+title: "Deploying Netdata to a New Server"
+domain: selfhosting
+category: monitoring
+tags: [netdata, monitoring, email, notifications, netdata-cloud, ubuntu, debian, n8n]
+status: published
+created: 2026-03-18
+updated: 2026-03-22
+---
+
+# Deploying Netdata to a New Server
+
+This covers the full Netdata setup for a new server in the fleet: install, email notification config, n8n webhook integration, and Netdata Cloud claim. Applies to Ubuntu/Debian servers.
+
+## 1. Install Prerequisites
+
+Install `jq` before anything else. It is required by the `custom_sender()` function in `health_alarm_notify.conf` to build the JSON payload sent to the n8n webhook. **If `jq` is missing, the webhook will fire with an empty body and n8n alert emails will have no information in them.**
+
+```bash
+apt install -y jq
+```
+
+Verify:
+
+```bash
+jq --version
+```
+
+## 2. Install Netdata
+
+Use the official kickstart script:
+
+```bash
+wget -O /tmp/netdata-install.sh https://get.netdata.cloud/kickstart.sh
+sh /tmp/netdata-install.sh --non-interactive --stable-channel --disable-telemetry
+```
+
+Verify it's running:
+
+```bash
+systemctl is-active netdata
+curl -s http://localhost:19999/api/v1/info | python3 -c "import sys,json; d=json.load(sys.stdin); print('Netdata', d['version'])"
+```
+
+## 3. Configure Email Notifications
+
+Copy the default config and set the three required values:
+
+```bash
+cp /usr/lib/netdata/conf.d/health_alarm_notify.conf /etc/netdata/health_alarm_notify.conf
+```
+
+Edit `/etc/netdata/health_alarm_notify.conf`:
+
+```ini
+EMAIL_SENDER="netdata@majorshouse.com"
+SEND_EMAIL="YES"
+DEFAULT_RECIPIENT_EMAIL="marcus@majorshouse.com"
+```
+
+Or apply with `sed` in one shot:
+
+```bash
+sed -i 's/^#\?EMAIL_SENDER=.*/EMAIL_SENDER="netdata@majorshouse.com"/' /etc/netdata/health_alarm_notify.conf
+sed -i 's/^#\?SEND_EMAIL=.*/SEND_EMAIL="YES"/' /etc/netdata/health_alarm_notify.conf
+sed -i 's/^#\?DEFAULT_RECIPIENT_EMAIL=.*/DEFAULT_RECIPIENT_EMAIL="marcus@majorshouse.com"/' /etc/netdata/health_alarm_notify.conf
+```
+
+Restart and test:
+
+```bash
+systemctl restart netdata
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test 2>&1 | grep -E '(OK|FAILED|email)'
+```
+
+You should see three `# OK` lines (WARNING → CRITICAL → CLEAR test cycle) and confirmation that email was sent to `marcus@majorshouse.com`.
+
+> [!note] Delivery via local Postfix
+> Email is relayed through the server's local Postfix instance. Ensure Postfix is installed and `/usr/sbin/sendmail` resolves.
+
+## 4. Configure n8n Webhook Notifications
+
+Copy the `health_alarm_notify.conf` from an existing server (e.g. majormail) which contains the `custom_sender()` function. This sends enriched JSON payloads to the n8n webhook at `https://n8n.majorshouse.com/webhook/netdata-alert`.
+
+> [!warning] jq required
+> The `custom_sender()` function uses `jq` to build the JSON payload. If `jq` is not installed, `payload` will be empty, curl will send `Content-Length: 0`, and n8n will produce alert emails with `Host: unknown`, blank alert/value fields, and `Status: UNKNOWN`. Always install `jq` first (Step 1).
+
+After deploying the config, run a test to confirm the webhook fires correctly:
+
+```bash
+systemctl restart netdata
+/usr/libexec/netdata/plugins.d/alarm-notify.sh test 2>&1 | grep -E '(custom|n8n|OK|FAILED)'
+```
+
+Verify in n8n that the latest execution shows a non-empty body with `hostname`, `alarm`, and `status` fields populated.
+
+## 5. Claim to Netdata Cloud
+
+Get the claim command from **Netdata Cloud → Space Settings → Nodes → Add Nodes**. It will look like:
+
+```bash
+wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh
+sh /tmp/netdata-kickstart.sh --stable-channel \
+  --claim-token <token> \
+  --claim-rooms <room-id> \
+  --claim-url https://app.netdata.cloud
+```
+
+Verify the claim was accepted:
+
+```bash
+cat /var/lib/netdata/cloud.d/claimed_id
+```
+
+A UUID will be present if claimed successfully. The node should appear in Netdata Cloud within ~60 seconds.
+
+## 6. Verify Alerts
+
+Check that no unexpected alerts are active after setup:
+
+```bash
+curl -s 'http://localhost:19999/api/v1/alarms?active' | python3 -c "
+import sys, json
+d = json.load(sys.stdin)
+active = [v for v in d.get('alarms', {}).values() if v.get('status') not in ('CLEAR', 'UNINITIALIZED', 'UNDEFINED')]
+print(f'{len(active)} active alert(s)')
+for v in active:
+    print(f'  [{v[\"status\"]}] {v[\"name\"]} on {v[\"chart\"]}')
+"
+```
+
+## Fleet-wide Alert Check
+
+To audit all servers at once (requires Tailscale SSH access):
+
+```bash
+for host in majorlab majorhome majormail majordiscord majortoot majorlinux tttpod dca teelia; do
+  echo "=== $host ==="
+  ssh root@$host "curl -s 'http://localhost:19999/api/v1/alarms?active' | python3 -c \
+    \"import sys,json; d=json.load(sys.stdin); active=[v for v in d.get('alarms',{}).values() if v.get('status') not in ('CLEAR','UNINITIALIZED','UNDEFINED')]; print(str(len(active))+' active')\""
+done
+```
+
+## Fleet-wide jq Audit
+
+To check that all servers with `custom_sender` have `jq` installed:
+
+```bash
+for host in majorlab majorhome majormail majordiscord majortoot majorlinux tttpod dca teelia; do
+  echo -n "=== $host: "
+  ssh -o ConnectTimeout=5 root@$host \
+    'has_cs=$(grep -l "custom_sender\|n8n.majorshouse.com" /etc/netdata/health_alarm_notify.conf 2>/dev/null | wc -l); has_jq=$(which jq 2>/dev/null && echo yes || echo NO); echo "custom_sender=$has_cs jq=$has_jq"'
+done
+```
+
+Any server showing `custom_sender=1 jq=NO` needs `apt install -y jq` immediately.
+
+## Related
+
+- [Tuning Netdata Web Log Alerts](tuning-netdata-web-log-alerts.md)
+- [Tuning Netdata Docker Health Alarms](netdata-docker-health-alarm-tuning.md)