From 1c17bdb60af941f67734f88377e2c981ec3bb222 Mon Sep 17 00:00:00 2001 From: MajorLinux Date: Fri, 8 May 2026 01:51:18 -0400 Subject: [PATCH] =?UTF-8?q?Add:=20Castopod=20federation=20=E2=80=94=20stal?= =?UTF-8?q?e=20cached=20avatar=20URL=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When a remote actor updates their avatar, Mastodon (Paperclip) deletes the old S3 object and stores only the new filename. Castopod 2.0.0 caches the URL of every federated actor in cp_fediverse_actors and never refetches, so its admin templates emit a dead link forever (the resulting S3 403 is anti-enumeration, hiding what is really a 404). Article documents the diagnosis pattern and three fixes (manual UPDATE, DELETE-and-refetch, bulk audit), plus the Mastodon-side query for sourcing the correct URL. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../castopod-stale-federated-avatar.md | 190 ++++++++++++++++++ SUMMARY.md | 1 + index.md | 6 +- 3 files changed, 195 insertions(+), 2 deletions(-) create mode 100644 05-troubleshooting/security/castopod-stale-federated-avatar.md diff --git a/05-troubleshooting/security/castopod-stale-federated-avatar.md b/05-troubleshooting/security/castopod-stale-federated-avatar.md new file mode 100644 index 0000000..ce86c3f --- /dev/null +++ b/05-troubleshooting/security/castopod-stale-federated-avatar.md @@ -0,0 +1,190 @@ +--- +title: "Castopod: Stale Federated Avatar URLs After Remote Profile Updates" +domain: troubleshooting +category: security +tags: [castopod, mastodon, fediverse, activitypub, s3, federation] +status: published +created: 2026-05-08 +updated: 2026-05-08 +--- + +# Castopod: Stale Federated Avatar URLs After Remote Profile Updates + +## πŸ›‘ Problem + +Your Castopod admin pages β€” most visibly the notifications list (`/cp-admin/podcasts//notifications`) β€” show broken avatars for federated actors. The browser dev tools (or a direct `curl -I`) on the avatar URL returns: + +``` +HTTP/1.1 403 Forbidden +Server: AmazonS3 +``` + +…with the response body: + +```xml + + AccessDenied + Access Denied + ... + +``` + +The hostname is the remote instance's S3 bucket (e.g. `s3.amazonaws.com//accounts/avatars/...`). Other actors in the same notifications list β€” those with avatars on Mastodon's own CDN, or on instances using path-stable storage β€” render fine. + +This article explains *why* the alarm code is misleading, *what's actually broken*, and how to fix it on Castopod. + +--- + +## πŸ”¬ Why "AccessDenied" is misleading + +S3 returns `403 AccessDenied` to anonymous requesters for **any** missing object β€” by design, as anti-enumeration. Anonymous users typically don't have `s3:ListBucket` permission on the bucket, so S3 deliberately can't tell them whether the key is missing or merely forbidden. Both cases produce the same 403. + +So when you see `403 AccessDenied` on a remote avatar URL, **the actual problem is almost always that the object no longer exists**. The bucket is fine; the file is gone. + +### Verifying that interpretation + +If you have access to the remote instance (or to S3 credentials for that bucket): + +```sh +aws s3api head-object --bucket --key accounts/avatars/.../.jpeg +``` + +If you see `An error occurred (404) when calling the HeadObject operation: Not Found`, the object is genuinely gone β€” and the upstream user has updated their avatar. + +--- + +## πŸ” What's actually broken + +Mastodon (and most ActivityPub servers using Paperclip-style storage) **deletes the old object** on avatar replacement and stores only the current filename in the DB. The remote instance is functioning normally β€” its current `` URL points to a different filename and serves correctly. + +Castopod 2.0.0 (verified up to `2.0.0-next.4`) **caches the avatar URL** of every federated actor in `cp_fediverse_actors.avatar_image_url` when it first sees activity from that actor β€” and never refetches. The admin templates (e.g. `themes/cp_admin/podcast/notifications.php`) emit that stored URL directly into ``. Once the upstream replaces the avatar: + +- Old object deleted β†’ S3 returns 403 to anonymous fetchers +- Castopod still renders the dead URL forever +- Every cached page using that template shows a broken image + +The same pattern applies to `cover_image_url` (header). + +--- + +## βœ… Fix + +You have three options, in increasing order of "this stays fixed." + +### Option 1 β€” Manual SQL update (one-shot) + +Recommended for one or two stale actors. Get the current URL from the upstream instance. + +If the upstream is your own Mastodon instance: + +```sh +sudo -u postgres psql mastodon_production -t -A \ + -c "SELECT id, avatar_file_name, header_file_name FROM accounts WHERE username=''" +``` + +Construct the canonical URL using the standard Paperclip path scheme. For an account ID like `109326168175475699`, the path is built by chunking the ID three digits at a time: + +``` +accounts/avatars/109/326/168/175/475/699/original/ +accounts/headers/109/326/168/175/475/699/original/ +``` + +Then UPDATE the Castopod row: + +```sh +mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME <<'SQL' +UPDATE cp_fediverse_actors +SET avatar_image_url = 'https:////accounts/avatars/109/326/168/175/475/699/original/.jpeg', + cover_image_url = 'https:////accounts/headers/109/326/168/175/475/699/original/.jpg', + updated_at = NOW() +WHERE username = '' + AND domain = ''; +SQL +``` + +Then clear the Castopod cache so any cached HTML rerenders: + +```sh +cd /var/www/html/castopod +sudo -u www-data php spark cache:clear +``` + +Verify: + +```sh +curl -sI 'https://' | head -1 # expect HTTP/1.1 200 OK +``` + +### Option 2 β€” Delete and let Castopod refetch + +For a one-shot self-healing fix, delete the actor row entirely: + +```sql +DELETE FROM cp_fediverse_actors WHERE username='' AND domain=''; +``` + +Castopod will repopulate the row from the next inbound activity from that actor (favourite, boost, mention, follow…). **Caveat β€” verify foreign-key cascades first:** `cp_fediverse_favourites`, `cp_fediverse_follows`, `cp_fediverse_posts`, and `cp_fediverse_notifications` all reference `actor_id`. Depending on the migration version, ON DELETE may cascade or restrict. Check with: + +```sh +mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME -e " + SELECT TABLE_NAME, CONSTRAINT_NAME, DELETE_RULE + FROM information_schema.REFERENTIAL_CONSTRAINTS + WHERE CONSTRAINT_SCHEMA = '$CP_DB_NAME' + AND REFERENCED_TABLE_NAME = 'cp_fediverse_actors'; +" +``` + +If deletes cascade, you'll lose the activity history attributed to that actor. Use Option 1 instead. + +### Option 3 β€” Bulk audit and update + +If multiple federated actors have likely-stale avatars (any old enough that an upstream user might have refreshed their profile picture), audit them all: + +```sh +mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME -BNe " + SELECT id, username, domain, avatar_image_url + FROM cp_fediverse_actors + WHERE avatar_image_url IS NOT NULL +" | while IFS=$'\t' read -r id user dom url; do + code=$(curl -s -o /dev/null -w "%{http_code}" "$url") + [ "$code" != "200" ] && echo "BROKEN $code $id $user@$dom $url" +done +``` + +For each broken row, fetch the upstream's current actor JSON and update from `icon.url` / `image.url`: + +```sh +curl -s -H 'Accept: application/activity+json' \ + "https:///users/" | jq '{icon, image}' +``` + +Then run the Option 1 SQL update with the fresh URLs. + +--- + +## πŸ§ͺ Why this isn't fixable on the upstream side + +Once the old object is deleted, you can't restore the URL without re-uploading bytes to the **exact original key** β€” which Mastodon won't do, because its DB only knows about the new filename. Trying to "fix" it on the Mastodon side means resurrecting a file Mastodon has no record of and that no fresh ActivityPub request would emit a URL for. The fix has to live on the consumer (Castopod) because Castopod is the one holding the stale reference. + +This applies to every federation consumer that caches URLs by reference rather than fetching bytes locally. Mastodon, Pleroma, Akkoma, and Misskey all cache the bytes; that's why they self-heal across remote avatar swaps. Castopod 2.0.0 currently does not. + +--- + +## πŸ›  Long-term mitigations + +This is a Castopod design issue worth raising upstream: +- Add a `last_refreshed_at` to `cp_fediverse_actors` and a worker that refetches actor JSON on a schedule. +- Or fetch and store avatars locally on first sight, the way Mastodon does. + +A `fediverse:refresh-actor` spark command would also let admins fix stale rows without writing SQL. + +If you have a recurring case (you update your Mastodon avatar often, and you also operate a Castopod instance under your own control), keep the Option 1 SQL handy as a one-liner. After your own avatar update, run it within minutes and the dead-URL window closes before it spreads to many cached pages. + +--- + +## πŸ“š References + +- Castopod source (`themes/cp_admin/podcast/notifications.php`) β€” uses `avatar_image_url` directly in `` +- AWS S3 anti-enumeration: `403` vs `404` is bucket-policy-dependent; see [GetObject β€” Permissions Required](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestPermissions) +- Mastodon Paperclip storage layout: `accounts/avatars/<3-digit chunks of account id>/original/` +- Related fix patterns: [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](netdata-web-log-successful-redirect-heavy-tuning.md) β€” shares the "the alarm is technically correct, but means something different than you think" theme diff --git a/SUMMARY.md b/SUMMARY.md index a33e956..dd96751 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -76,6 +76,7 @@ updated: 2026-05-08T01:08 * [Fail2ban & UFW Rule Bloat Cleanup](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md) * [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) * [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md) + * [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md) * [Nextcloud AIO Unhealthy 20h After Nightly Update](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md) * [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md) * [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md) diff --git a/index.md b/index.md index da88f8e..8747c10 100644 --- a/index.md +++ b/index.md @@ -7,7 +7,7 @@ updated: 2026-05-08T01:08 > A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin. > > **Last updated:** 2026-05-08 -> **Article count:** 107 +> **Article count:** 108 ## Domains @@ -17,7 +17,7 @@ updated: 2026-05-08T01:08 | 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 39 | | πŸ”“ Open Source Tools | `03-opensource/` | 10 | | πŸŽ™οΈ Streaming & Podcasting | `04-streaming/` | 2 | -| πŸ”§ General Troubleshooting | `05-troubleshooting/` | 44 | +| πŸ”§ General Troubleshooting | `05-troubleshooting/` | 45 | --- @@ -201,6 +201,7 @@ updated: 2026-05-08T01:08 - [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md) - [Custom Fail2ban Jail: Apache Directory Scanning & Junk Methods](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md) - [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md) +- [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md) ### Storage - [mdadm RAID Recovery After USB Hub Disconnect](05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md) @@ -215,6 +216,7 @@ updated: 2026-05-08T01:08 | Date | Article | Domain | |---|---|---| +| 2026-05-08 | [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md) | Troubleshooting | | 2026-05-08 | [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md) | Troubleshooting | | 2026-05-07 | [Mastodon β€” The `--prune-profiles` Trap and How to Recover](02-selfhosting/services/mastodon-prune-profiles-trap.md) | Self-Hosting | | 2026-05-02 | [WSL2 Backup via PowerShell Scheduled Task](01-linux/distro-specific/wsl2-backup-powershell.md) | Linux |