Files

MajorLinux fb2e3f6168 wiki: add SELinux AVC chart, enriched alerts, new server setup, and pending articles; update indexes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-03-27 03:34:33 -04:00

5.5 KiB

Raw Blame History

title, domain, category, tags, status, created, updated

title

domain

Deploying Netdata to a New Server

This covers the full Netdata setup for a new server in the fleet: install, email notification config, n8n webhook integration, and Netdata Cloud claim. Applies to Ubuntu/Debian servers.

1. Install Prerequisites

Install jq before anything else. It is required by the custom_sender() function in health_alarm_notify.conf to build the JSON payload sent to the n8n webhook. If jq is missing, the webhook will fire with an empty body and n8n alert emails will have no information in them.

apt install -y jq

Verify:

jq --version

2. Install Netdata

Use the official kickstart script:

wget -O /tmp/netdata-install.sh https://get.netdata.cloud/kickstart.sh
sh /tmp/netdata-install.sh --non-interactive --stable-channel --disable-telemetry

Verify it's running:

systemctl is-active netdata
curl -s http://localhost:19999/api/v1/info | python3 -c "import sys,json; d=json.load(sys.stdin); print('Netdata', d['version'])"

3. Configure Email Notifications

Copy the default config and set the three required values:

cp /usr/lib/netdata/conf.d/health_alarm_notify.conf /etc/netdata/health_alarm_notify.conf

Edit /etc/netdata/health_alarm_notify.conf:

EMAIL_SENDER="netdata@majorshouse.com"
SEND_EMAIL="YES"
DEFAULT_RECIPIENT_EMAIL="marcus@majorshouse.com"

Or apply with sed in one shot:

sed -i 's/^#\?EMAIL_SENDER=.*/EMAIL_SENDER="netdata@majorshouse.com"/' /etc/netdata/health_alarm_notify.conf
sed -i 's/^#\?SEND_EMAIL=.*/SEND_EMAIL="YES"/' /etc/netdata/health_alarm_notify.conf
sed -i 's/^#\?DEFAULT_RECIPIENT_EMAIL=.*/DEFAULT_RECIPIENT_EMAIL="marcus@majorshouse.com"/' /etc/netdata/health_alarm_notify.conf

Restart and test:

systemctl restart netdata
/usr/libexec/netdata/plugins.d/alarm-notify.sh test 2>&1 | grep -E '(OK|FAILED|email)'

You should see three # OK lines (WARNING → CRITICAL → CLEAR test cycle) and confirmation that email was sent to marcus@majorshouse.com.

[!note] Delivery via local Postfix Email is relayed through the server's local Postfix instance. Ensure Postfix is installed and /usr/sbin/sendmail resolves.

4. Configure n8n Webhook Notifications

Copy the health_alarm_notify.conf from an existing server (e.g. majormail) which contains the custom_sender() function. This sends enriched JSON payloads to the n8n webhook at https://n8n.majorshouse.com/webhook/netdata-alert.

[!warning] jq required The custom_sender() function uses jq to build the JSON payload. If jq is not installed, payload will be empty, curl will send Content-Length: 0, and n8n will produce alert emails with Host: unknown, blank alert/value fields, and Status: UNKNOWN. Always install jq first (Step 1).

After deploying the config, run a test to confirm the webhook fires correctly:

systemctl restart netdata
/usr/libexec/netdata/plugins.d/alarm-notify.sh test 2>&1 | grep -E '(custom|n8n|OK|FAILED)'

Verify in n8n that the latest execution shows a non-empty body with hostname, alarm, and status fields populated.

5. Claim to Netdata Cloud

Get the claim command from Netdata Cloud → Space Settings → Nodes → Add Nodes. It will look like:

wget -O /tmp/netdata-kickstart.sh https://get.netdata.cloud/kickstart.sh
sh /tmp/netdata-kickstart.sh --stable-channel \
  --claim-token <token> \
  --claim-rooms <room-id> \
  --claim-url https://app.netdata.cloud

Verify the claim was accepted:

cat /var/lib/netdata/cloud.d/claimed_id

A UUID will be present if claimed successfully. The node should appear in Netdata Cloud within ~60 seconds.

6. Verify Alerts

Check that no unexpected alerts are active after setup:

curl -s 'http://localhost:19999/api/v1/alarms?active' | python3 -c "
import sys, json
d = json.load(sys.stdin)
active = [v for v in d.get('alarms', {}).values() if v.get('status') not in ('CLEAR', 'UNINITIALIZED', 'UNDEFINED')]
print(f'{len(active)} active alert(s)')
for v in active:
    print(f'  [{v[\"status\"]}] {v[\"name\"]} on {v[\"chart\"]}')
"

Fleet-wide Alert Check

To audit all servers at once (requires Tailscale SSH access):

for host in majorlab majorhome majormail majordiscord majortoot majorlinux tttpod dca teelia; do
  echo "=== $host ==="
  ssh root@$host "curl -s 'http://localhost:19999/api/v1/alarms?active' | python3 -c \
    \"import sys,json; d=json.load(sys.stdin); active=[v for v in d.get('alarms',{}).values() if v.get('status') not in ('CLEAR','UNINITIALIZED','UNDEFINED')]; print(str(len(active))+' active')\""
done

Fleet-wide jq Audit

To check that all servers with custom_sender have jq installed:

for host in majorlab majorhome majormail majordiscord majortoot majorlinux tttpod dca teelia; do
  echo -n "=== $host: "
  ssh -o ConnectTimeout=5 root@$host \
    'has_cs=$(grep -l "custom_sender\|n8n.majorshouse.com" /etc/netdata/health_alarm_notify.conf 2>/dev/null | wc -l); has_jq=$(which jq 2>/dev/null && echo yes || echo NO); echo "custom_sender=$has_cs jq=$has_jq"'
done

Any server showing custom_sender=1 jq=NO needs apt install -y jq immediately.

5.5 KiB Raw Blame History