160 lines
6.0 KiB
Markdown
160 lines
6.0 KiB
Markdown
# Netdata → n8n Enriched Alert Emails
|
|
|
|
**Status:** Live across all MajorsHouse fleet servers as of 2026-03-21
|
|
|
|
Replaces Netdata's plain-text alert emails with rich HTML emails that include a plain-English explanation, a suggested remediation command, and a direct link to the relevant MajorWiki article.
|
|
|
|
---
|
|
|
|
## How It Works
|
|
|
|
```
|
|
Netdata alarm fires
|
|
→ custom_sender() in health_alarm_notify.conf
|
|
→ POST JSON payload to n8n webhook
|
|
→ Code node enriches with suggestion + wiki link
|
|
→ Send Email node sends HTML email via SMTP
|
|
→ Respond node returns 200 OK
|
|
```
|
|
|
|
---
|
|
|
|
## n8n Workflow
|
|
|
|
**Name:** Netdata Enriched Alerts
|
|
**URL:** https://n8n.majorshouse.com
|
|
**Webhook endpoint:** `POST https://n8n.majorshouse.com/webhook/netdata-alert`
|
|
**Workflow ID:** `a1b2c3d4-aaaa-bbbb-cccc-000000000001`
|
|
|
|
### Nodes
|
|
|
|
1. **Netdata Webhook** — receives POST from Netdata's `custom_sender()`
|
|
2. **Enrich Alert** — Code node; matches alarm/chart/family to enrichment table, builds HTML email body in `$json.emailBody`
|
|
3. **Send Enriched Email** — sends via SMTP port 465 (SMTP account 2), from `netdata@majorshouse.com` to `marcus@majorshouse.com`
|
|
4. **Respond OK** — returns `ok` with HTTP 200 to Netdata
|
|
|
|
### Enrichment Keys
|
|
|
|
The Code node matches on `alarm`, `chart`, or `family` field (case-insensitive substring):
|
|
|
|
| Key | Title | Wiki Article | Notes |
|
|
|-----|-------|-------------|-------|
|
|
| `disk_space` | Disk Space Alert | snapraid-mergerfs-setup | |
|
|
| `ram` | Memory Alert | managing-linux-services-systemd-ansible | |
|
|
| `cpu` | CPU Alert | managing-linux-services-systemd-ansible | |
|
|
| `load` | Load Average Alert | managing-linux-services-systemd-ansible | |
|
|
| `net` | Network Alert | tailscale-homelab-remote-access | |
|
|
| `docker` | Docker Container Alert | debugging-broken-docker-containers | |
|
|
| `web_log` | Web Log Alert | tuning-netdata-web-log-alerts | Hostname-aware suggestion (see below) |
|
|
| `health` | Docker Health Alarm | netdata-docker-health-alarm-tuning | |
|
|
| `mdstat` | RAID Array Alert | mdadm-usb-hub-disconnect-recovery | |
|
|
| `systemd` | Systemd Service Alert | docker-caddy-selinux-post-reboot-recovery | |
|
|
| _(no match)_ | Server Alert | netdata-new-server-setup | |
|
|
|
|
> [!info] web_log hostname-aware suggestion (updated 2026-03-24)
|
|
> The `web_log` suggestion branches on `hostname` in the Code node:
|
|
> - **`majorlab`** → Check `docker logs caddy` (Caddy reverse proxy)
|
|
> - **`teelia`, `majorlinux`, `dca`** → Check Apache logs + Fail2ban jail status
|
|
> - **other** → Generic web server log guidance
|
|
|
|
---
|
|
|
|
## Netdata Configuration
|
|
|
|
### Config File Locations
|
|
|
|
| Server | Path |
|
|
|--------|------|
|
|
| majorhome, majormail, majordiscord, tttpod, teelia | `/etc/netdata/health_alarm_notify.conf` |
|
|
| majorlinux, majortoot, dca | `/usr/lib/netdata/conf.d/health_alarm_notify.conf` |
|
|
|
|
### Required Settings
|
|
|
|
```bash
|
|
DEFAULT_RECIPIENT_CUSTOM="n8n"
|
|
role_recipients_custom[sysadmin]="${DEFAULT_RECIPIENT_CUSTOM}"
|
|
```
|
|
|
|
### custom_sender() Function
|
|
|
|
```bash
|
|
custom_sender() {
|
|
local to="${1}"
|
|
local payload
|
|
payload=$(jq -n \
|
|
--arg hostname "${host}" \
|
|
--arg alarm "${name}" \
|
|
--arg chart "${chart}" \
|
|
--arg family "${family}" \
|
|
--arg status "${status}" \
|
|
--arg old_status "${old_status}" \
|
|
--arg value "${value_string}" \
|
|
--arg units "${units}" \
|
|
--arg info "${info}" \
|
|
--arg alert_url "${goto_url}" \
|
|
--arg severity "${severity}" \
|
|
--arg raised_for "${raised_for}" \
|
|
--arg total_warnings "${total_warnings}" \
|
|
--arg total_critical "${total_critical}" \
|
|
'{hostname:$hostname,alarm:$alarm,chart:$chart,family:$family,status:$status,old_status:$old_status,value:$value,units:$units,info:$info,alert_url:$alert_url,severity:$severity,raised_for:$raised_for,total_warnings:$total_warnings,total_critical:$total_critical}')
|
|
local httpcode
|
|
httpcode=$(docurl -s -o /dev/null -w "%{http_code}" \
|
|
-X POST \
|
|
-H "Content-Type: application/json" \
|
|
-d "${payload}" \
|
|
"https://n8n.majorshouse.com/webhook/netdata-alert")
|
|
if [ "${httpcode}" = "200" ]; then
|
|
info "sent enriched notification to n8n for ${status} of ${host}.${name}"
|
|
sent=$((sent + 1))
|
|
else
|
|
error "failed to send notification to n8n, HTTP code: ${httpcode}"
|
|
fi
|
|
}
|
|
```
|
|
|
|
!!! note "jq required"
|
|
The `custom_sender()` function requires `jq` to be installed. Verify with `which jq` on each server.
|
|
|
|
---
|
|
|
|
## Deploying to a New Server
|
|
|
|
```bash
|
|
# 1. Find the config file
|
|
find /etc/netdata /usr/lib/netdata -name health_alarm_notify.conf 2>/dev/null
|
|
|
|
# 2. Edit it — add the two lines and the custom_sender() function above
|
|
|
|
# 3. Test connectivity from the server
|
|
curl -s -o /dev/null -w "%{http_code}" \
|
|
-X POST https://n8n.majorshouse.com/webhook/netdata-alert \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"hostname":"test","alarm":"disk_space._","status":"WARNING"}'
|
|
# Expected: 200
|
|
|
|
# 4. Restart Netdata
|
|
systemctl restart netdata
|
|
|
|
# 5. Send a test alarm
|
|
/usr/libexec/netdata/plugins.d/alarm-notify.sh test custom
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
**Emails not arriving — check n8n execution log:**
|
|
Go to https://n8n.majorshouse.com → open "Netdata Enriched Alerts" → Executions tab. Look for `error` status entries.
|
|
|
|
**Email body empty:**
|
|
The Send Email node's HTML field must be `={{ $json.emailBody }}`. Shell variable expansion can silently strip `$json` if the workflow is patched via inline SSH commands — always use a Python script file.
|
|
|
|
**`000` curl response from a server:**
|
|
Usually a timeout, not a DNS or connection failure. Re-test with `--max-time 30`.
|
|
|
|
**`custom_sender()` syntax error in Netdata logs:**
|
|
Bash heredocs don't work inside sourced config files. Use `jq -n --arg ...` as shown above — no heredocs.
|
|
|
|
**n8n `N8N_TRUST_PROXY` must be set:**
|
|
Without `N8N_TRUST_PROXY=true` in the Docker environment, Caddy's `X-Forwarded-For` header causes n8n's rate limiter to abort requests before parsing the body. Set in `/opt/n8n/compose.yml`.
|