From 4599ed607ce5641b05fa4a17977920b42baee303 Mon Sep 17 00:00:00 2001 From: majorlinux Date: Fri, 19 Jun 2026 10:05:06 -0400 Subject: [PATCH] wiki: add restic + B2 fleet backups runbook Architecture, per-engine DB dump patterns, restore procedure, add-a-host, and gotchas (RESTIC_CACHE_DIR/$HOME, missing sqlite3, docker dump env vars, delete-capable B2 key). Linked in SUMMARY under storage-backup. --- .../storage-backup/restic-b2-fleet-backups.md | 136 ++++++++++++++++++ SUMMARY.md | 3 +- 2 files changed, 138 insertions(+), 1 deletion(-) create mode 100644 02-selfhosting/storage-backup/restic-b2-fleet-backups.md diff --git a/02-selfhosting/storage-backup/restic-b2-fleet-backups.md b/02-selfhosting/storage-backup/restic-b2-fleet-backups.md new file mode 100644 index 0000000..6694870 --- /dev/null +++ b/02-selfhosting/storage-backup/restic-b2-fleet-backups.md @@ -0,0 +1,136 @@ +--- +title: "App-Consistent Fleet Backups with restic + Backblaze B2" +domain: selfhosting +category: storage-backup +tags: [restic, backblaze, b2, backup, ansible, systemd, postgresql, mysql, sqlite, docker, disaster-recovery] +status: published +created: 2026-06-19 +updated: 2026-06-19 +--- + +# App-Consistent Fleet Backups with restic + Backblaze B2 + +A repeatable pattern for backing up a mixed fleet (Ubuntu + Fedora, VPS + homelab, bare services + Docker) to Backblaze B2 with [restic](https://restic.net) — encrypted, deduplicated, and **app-consistent** (databases are dumped before the snapshot, not copied live). Driven by Ansible and a per-host `systemd` timer. + +## The Short Answer + +Per host, nightly: **dump every database to a staging dir → `restic backup` that staging dir plus the data paths → apply retention → wipe staging.** A monthly timer runs `restic prune`. Anything that fails emails the admin. One B2 bucket holds a separate repo per host at `b2::`. + +Retention is `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` (~6 months of history). + +## Why dump databases first + +Copying a live database's files (`/var/lib/mysql`, a running SQLite file, a Postgres data dir) gives you a *crash-consistent* copy at best — restorable only if you're lucky. Logical dumps are guaranteed consistent: + +- **MySQL / MariaDB:** `mysqldump --single-transaction --routines --triggers --databases ` +- **PostgreSQL:** `pg_dump -Fc ` (custom format) via the `postgres` system user (peer auth) +- **SQLite:** `sqlite3 ".backup ''"` — uses the online backup API, safe against a running writer +- **Dockerized DBs:** `docker exec sh -c ''`, letting the container's own shell expand its root-password env var + +restic then backs up the dump files (which dedupe beautifully — only the changed blocks upload each night). + +## Repository layout + +- **One private B2 bucket** (e.g. `majorshouse-backups`). +- **One repo per host:** `b2:majorshouse-backups:`. +- The application key needs **read + write + delete** for the bucket. restic deletes objects during `forget`/`prune`, so a pure *append-only* key will break retention. (True append-only requires splitting `forget`/`prune` onto a separate maintenance key — a worthwhile hardening step, but not the default.) +- Credentials live in an `EnvironmentFile` (`/etc/restic/restic-env`, mode `0600`, root): `RESTIC_REPOSITORY`, `RESTIC_PASSWORD`, `B2_ACCOUNT_ID`, `B2_ACCOUNT_KEY`. + +## The backup script (shape) + +```bash +set -uo pipefail +STAGING=/var/backups/restic-staging +rm -rf "$STAGING"; mkdir -p "$STAGING"; chmod 700 "$STAGING" + +# per-engine dumps into $STAGING ... +mysqldump --single-transaction --routines --triggers --databases wordpress > "$STAGING/mysql-wordpress.sql" +sudo -u postgres pg_dump -Fc mastodon_production > "$STAGING/pg-mastodon_production.dump" +sqlite3 /opt/phantombot/config/phantombot.db ".backup '$STAGING/sqlite-phantombot.db'" + +restic backup --tag fleet-backup --host "$(hostname -s)" \ + "$STAGING" /var/www /etc/letsencrypt --exclude /path/to/already-offsite/media + +restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 6 +rm -rf "$STAGING" +``` + +Wrap each step so a failure mails the admin and aborts (don't silently back up a half-state). On hosts where the `mail` CLI is absent, pipe a message to `/usr/sbin/sendmail -t` instead. + +## systemd units + +A oneshot service + a timer. Stagger `OnCalendar` per host to spread B2 load, and **always set `RESTIC_CACHE_DIR`** (see Gotchas): + +```ini +# restic-backup.service +[Service] +Type=oneshot +EnvironmentFile=/etc/restic/restic-env +Environment=RESTIC_CACHE_DIR=/var/cache/restic +ExecStart=/usr/local/sbin/restic-backup.sh +Nice=10 +IOSchedulingClass=idle +``` + +```ini +# restic-backup.timer +[Timer] +OnCalendar=*-*-* 02:30:00 +RandomizedDelaySec=20m +Persistent=true +[Install] +WantedBy=timers.target +``` + +A second `restic-prune.timer` runs `restic prune` monthly (`OnCalendar=*-*-01 04:00:00`). + +## Restore procedure + +The whole point. From the target host (or any host with the repo creds): + +```bash +# load repo + B2 creds without echoing them +set -a; . /etc/restic/restic-env; set +a + +restic snapshots # list; note the snapshot ID or use 'latest' + +# restore specific paths to a scratch dir (never restore in place blindly) +restic restore latest --target /tmp/restore \ + --include /var/backups/restic-staging \ + --include /var/www/html/wp-config.php + +# verify before doing anything with it +ls -la /tmp/restore/var/backups/restic-staging/ +head -1 /tmp/restore/var/backups/restic-staging/mysql-wordpress.sql # "-- MySQL dump 10.13 ..." +``` + +To recover a database, restore the dump then load it: `mysql < mysql-.sql`, `pg_restore -d pg-.dump`, or copy the SQLite file back. **Test restores periodically** — a backup you've never restored is a hope, not a backup. Restore the highest-stakes data (password manager, mail) first in any drill. + +## Adding a host + +1. Add it to the `backups` inventory group. +2. Give it a `host_vars` scope — which DBs to dump and which paths to back up: + + ```yaml + restic_backup_oncalendar: "*-*-* 02:40:00" # stagger + restic_mysql_dbs: [castopod_db] + restic_paths: [/var/www/html/castopod] + restic_excludes: [/var/www/html/castopod/public/media] # already offsite + ``` +3. Run the playbook against that host. The role installs restic, deploys the script + units, `restic init`s the repo if absent, and enables the timers. + +## Gotchas & Notes + +- **`RESTIC_CACHE_DIR` is mandatory under systemd.** systemd services run with no `$HOME`, so restic can't find its cache and warns *"unable to locate cache directory: neither $XDG_CACHE_HOME nor $HOME are defined"* — and re-reads **every file** each run (no incremental). Point it at `/var/cache/restic` in the unit. +- **`sqlite3` may not be installed.** A host that runs a SQLite-backed app (e.g. a bot) often lacks the `sqlite3`/`sqlite` CLI. Install it where `restic_sqlite_paths` is set, or the `.backup` step fails. +- **Docker DB password env-var names vary.** Don't assume: the MariaDB image may use `MYSQL_ROOT_PASSWORD` (not `MARIADB_ROOT_PASSWORD`), and a Postgres container's superuser is whatever `POSTGRES_USER` is set to — reference `"$POSTGRES_USER"` rather than hardcoding `postgres`. Check with `docker exec sh -c 'env | grep -oE "^(MYSQL|MARIADB|POSTGRES)_[A-Z_]*"'` (name only). +- **B2 key needs delete capability.** Otherwise `forget`/`prune` fail. Scope the key to the bucket; reach for per-host `namePrefix`-restricted keys for blast-radius isolation. +- **Exclude data that's already offsite.** Media already synced to object storage (S3/B2 via the app or `rclone`) should be `--exclude`d so you don't pay to store it twice. +- **First upload is slow, the rest are fast.** The initial snapshot reads and uploads everything; subsequent runs only ship changed blocks. For a large first run, fire it detached and watch from a transient unit that emails you on completion. +- **Keep secrets out of git.** The repo password and B2 key belong in an Ansible vault (committed encrypted), referenced into the role — never in plaintext vars. + +## See Also + +- [rsync Backup Patterns](rsync-backup-patterns.md) +- [SnapRAID & MergerFS Storage Setup](../../01-linux/storage/snapraid-mergerfs-setup.md) +- [restic documentation](https://restic.readthedocs.io) diff --git a/SUMMARY.md b/SUMMARY.md index 13bc6d8..dfa413c 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -1,6 +1,6 @@ --- created: 2026-04-02T16:03 -updated: 2026-05-15T09:00 +updated: 2026-06-19T10:05 --- * [Home](index.md) * [Linux & Sysadmin](01-linux/index.md) @@ -31,6 +31,7 @@ updated: 2026-05-15T09:00 * [AWS S3 Cost Management](02-selfhosting/cloud/aws-s3-cost-management.md) * [VPS Migration Baseline Checklist](02-selfhosting/cloud/vps-migration-baseline-checklist.md) * [rsync Backup Patterns](02-selfhosting/storage-backup/rsync-backup-patterns.md) + * [Fleet Backups with restic + B2](02-selfhosting/storage-backup/restic-b2-fleet-backups.md) * [Tuning Netdata Web Log Alerts](02-selfhosting/monitoring/tuning-netdata-web-log-alerts.md) * [Tuning Netdata Docker Health Alarms](02-selfhosting/monitoring/netdata-docker-health-alarm-tuning.md) * [Deploying Netdata to a New Server](02-selfhosting/monitoring/netdata-new-server-setup.md)