wiki: add restic + B2 fleet backups runbook

Architecture, per-engine DB dump patterns, restore procedure, add-a-host,
and gotchas (RESTIC_CACHE_DIR/$HOME, missing sqlite3, docker dump env vars,
delete-capable B2 key). Linked in SUMMARY under storage-backup.

02-selfhosting/storage-backup/restic-b2-fleet-backups.md

@@ -0,0 +1,136 @@
+---
+title: "App-Consistent Fleet Backups with restic + Backblaze B2"
+domain: selfhosting
+category: storage-backup
+tags: [restic, backblaze, b2, backup, ansible, systemd, postgresql, mysql, sqlite, docker, disaster-recovery]
+status: published
+created: 2026-06-19
+updated: 2026-06-19
+---
+
+# App-Consistent Fleet Backups with restic + Backblaze B2
+
+A repeatable pattern for backing up a mixed fleet (Ubuntu + Fedora, VPS + homelab, bare services + Docker) to Backblaze B2 with [restic](https://restic.net) — encrypted, deduplicated, and **app-consistent** (databases are dumped before the snapshot, not copied live). Driven by Ansible and a per-host `systemd` timer.
+
+## The Short Answer
+
+Per host, nightly: **dump every database to a staging dir → `restic backup` that staging dir plus the data paths → apply retention → wipe staging.** A monthly timer runs `restic prune`. Anything that fails emails the admin. One B2 bucket holds a separate repo per host at `b2:<bucket>:<hostname>`.
+
+Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` (~6 months of history).
+
+## Why dump databases first
+
+Copying a live database's files (`/var/lib/mysql`, a running SQLite file, a Postgres data dir) gives you a *crash-consistent* copy at best — restorable only if you're lucky. Logical dumps are guaranteed consistent:
+
+- **MySQL / MariaDB:** `mysqldump --single-transaction --routines --triggers --databases <db>`
+- **PostgreSQL:** `pg_dump -Fc <db>` (custom format) via the `postgres` system user (peer auth)
+- **SQLite:** `sqlite3 <file> ".backup '<out>'"` — uses the online backup API, safe against a running writer
+- **Dockerized DBs:** `docker exec <container> sh -c '<dump cmd>'`, letting the container's own shell expand its root-password env var
+
+restic then backs up the dump files (which dedupe beautifully — only the changed blocks upload each night).
+
+## Repository layout
+
+- **One private B2 bucket** (e.g. `majorshouse-backups`).
+- **One repo per host:** `b2:majorshouse-backups:<hostname>`.
+- The application key needs **read + write + delete** for the bucket. restic deletes objects during `forget`/`prune`, so a pure *append-only* key will break retention. (True append-only requires splitting `forget`/`prune` onto a separate maintenance key — a worthwhile hardening step, but not the default.)
+- Credentials live in an `EnvironmentFile` (`/etc/restic/restic-env`, mode `0600`, root): `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`, `B2_ACCOUNT_ID`, `B2_ACCOUNT_KEY`.
+
+## The backup script (shape)
+
+```bash
+set -uo pipefail
+STAGING=/var/backups/restic-staging
+rm -rf "$STAGING"; mkdir -p "$STAGING"; chmod 700 "$STAGING"
+
+# per-engine dumps into $STAGING ...
+mysqldump --single-transaction --routines --triggers --databases wordpress > "$STAGING/mysql-wordpress.sql"
+sudo -u postgres pg_dump -Fc mastodon_production            > "$STAGING/pg-mastodon_production.dump"
+sqlite3 /opt/phantombot/config/phantombot.db ".backup '$STAGING/sqlite-phantombot.db'"
+
+restic backup --tag fleet-backup --host "$(hostname -s)" \
+  "$STAGING" /var/www /etc/letsencrypt --exclude /path/to/already-offsite/media
+
+restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6
+rm -rf "$STAGING"
+```
+
+Wrap each step so a failure mails the admin and aborts (don't silently back up a half-state). On hosts where the `mail` CLI is absent, pipe a message to `/usr/sbin/sendmail -t` instead.
+
+## systemd units
+
+A oneshot service + a timer. Stagger `OnCalendar` per host to spread B2 load, and **always set `RESTIC_CACHE_DIR`** (see Gotchas):
+
+```ini
+# restic-backup.service
+[Service]
+Type=oneshot
+EnvironmentFile=/etc/restic/restic-env
+Environment=RESTIC_CACHE_DIR=/var/cache/restic
+ExecStart=/usr/local/sbin/restic-backup.sh
+Nice=10
+IOSchedulingClass=idle
+```
+
+```ini
+# restic-backup.timer
+[Timer]
+OnCalendar=*-*-* 02:30:00
+RandomizedDelaySec=20m
+Persistent=true
+[Install]
+WantedBy=timers.target
+```
+
+A second `restic-prune.timer` runs `restic prune` monthly (`OnCalendar=*-*-01 04:00:00`).
+
+## Restore procedure
+
+The whole point. From the target host (or any host with the repo creds):
+
+```bash
+# load repo + B2 creds without echoing them
+set -a; . /etc/restic/restic-env; set +a
+
+restic snapshots                      # list; note the snapshot ID or use 'latest'
+
+# restore specific paths to a scratch dir (never restore in place blindly)
+restic restore latest --target /tmp/restore \
+  --include /var/backups/restic-staging \
+  --include /var/www/html/wp-config.php
+
+# verify before doing anything with it
+ls -la /tmp/restore/var/backups/restic-staging/
+head -1 /tmp/restore/var/backups/restic-staging/mysql-wordpress.sql   # "-- MySQL dump 10.13 ..."
+```
+
+To recover a database, restore the dump then load it: `mysql <db> < mysql-<db>.sql`, `pg_restore -d <db> pg-<db>.dump`, or copy the SQLite file back. **Test restores periodically** — a backup you've never restored is a hope, not a backup. Restore the highest-stakes data (password manager, mail) first in any drill.
+
+## Adding a host
+
+1. Add it to the `backups` inventory group.
+2. Give it a `host_vars` scope — which DBs to dump and which paths to back up:
+
+   ```yaml
+   restic_backup_oncalendar: "*-*-* 02:40:00"   # stagger
+   restic_mysql_dbs: [castopod_db]
+   restic_paths: [/var/www/html/castopod]
+   restic_excludes: [/var/www/html/castopod/public/media]   # already offsite
+   ```
+3. Run the playbook against that host. The role installs restic, deploys the script + units, `restic init`s the repo if absent, and enables the timers.
+
+## Gotchas & Notes
+
+- **`RESTIC_CACHE_DIR` is mandatory under systemd.** systemd services run with no `$HOME`, so restic can't find its cache and warns *"unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are defined"* — and re-reads **every file** each run (no incremental). Point it at `/var/cache/restic` in the unit.
+- **`sqlite3` may not be installed.** A host that runs a SQLite-backed app (e.g. a bot) often lacks the `sqlite3`/`sqlite` CLI. Install it where `restic_sqlite_paths` is set, or the `.backup` step fails.
+- **Docker DB password env-var names vary.** Don't assume: the MariaDB image may use `MYSQL_ROOT_PASSWORD` (not `MARIADB_ROOT_PASSWORD`), and a Postgres container's superuser is whatever `POSTGRES_USER` is set to — reference `"$POSTGRES_USER"` rather than hardcoding `postgres`. Check with `docker exec <c> sh -c 'env | grep -oE "^(MYSQL|MARIADB|POSTGRES)_[A-Z_]*"'` (name only).
+- **B2 key needs delete capability.** Otherwise `forget`/`prune` fail. Scope the key to the bucket; reach for per-host `namePrefix`-restricted keys for blast-radius isolation.
+- **Exclude data that's already offsite.** Media already synced to object storage (S3/B2 via the app or `rclone`) should be `--exclude`d so you don't pay to store it twice.
+- **First upload is slow, the rest are fast.** The initial snapshot reads and uploads everything; subsequent runs only ship changed blocks. For a large first run, fire it detached and watch from a transient unit that emails you on completion.
+- **Keep secrets out of git.** The repo password and B2 key belong in an Ansible vault (committed encrypted), referenced into the role — never in plaintext vars.
+
+## See Also
+
+- [rsync Backup Patterns](rsync-backup-patterns.md)
+- [SnapRAID & MergerFS Storage Setup](../../01-linux/storage/snapraid-mergerfs-setup.md)
+- [restic documentation](https://restic.readthedocs.io)

SUMMARY.md

@@ -1,6 +1,6 @@
 ---
 created: 2026-04-02T16:03
-updated: 2026-05-15T09:00
+updated: 2026-06-19T10:05
 ---
 * [Home](index.md)
 * [Linux & Sysadmin](01-linux/index.md)
@@ -31,6 +31,7 @@ updated: 2026-05-15T09:00
     * [AWS S3 Cost Management](02-selfhosting/cloud/aws-s3-cost-management.md)
     * [VPS Migration Baseline Checklist](02-selfhosting/cloud/vps-migration-baseline-checklist.md)
     * [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md)
+    * [Fleet Backups with restic + B2](02-selfhosting/storage-backup/restic-b2-fleet-backups.md)
     * [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md)
     * [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md)
     * [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md)