Add: Castopod federation — stale cached avatar URL fix
When a remote actor updates their avatar, Mastodon (Paperclip) deletes the old S3 object and stores only the new filename. Castopod 2.0.0 caches the URL of every federated actor in cp_fediverse_actors and never refetches, so its admin templates emit a dead link forever (the resulting S3 403 is anti-enumeration, hiding what is really a 404). Article documents the diagnosis pattern and three fixes (manual UPDATE, DELETE-and-refetch, bulk audit), plus the Mastodon-side query for sourcing the correct URL. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
393df3cc45
commit
1c17bdb60a
3 changed files with 195 additions and 2 deletions
190
05-troubleshooting/security/castopod-stale-federated-avatar.md
Normal file
190
05-troubleshooting/security/castopod-stale-federated-avatar.md
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
---
|
||||
title: "Castopod: Stale Federated Avatar URLs After Remote Profile Updates"
|
||||
domain: troubleshooting
|
||||
category: security
|
||||
tags: [castopod, mastodon, fediverse, activitypub, s3, federation]
|
||||
status: published
|
||||
created: 2026-05-08
|
||||
updated: 2026-05-08
|
||||
---
|
||||
|
||||
# Castopod: Stale Federated Avatar URLs After Remote Profile Updates
|
||||
|
||||
## 🛑 Problem
|
||||
|
||||
Your Castopod admin pages — most visibly the notifications list (`/cp-admin/podcasts/<id>/notifications`) — show broken avatars for federated actors. The browser dev tools (or a direct `curl -I`) on the avatar URL returns:
|
||||
|
||||
```
|
||||
HTTP/1.1 403 Forbidden
|
||||
Server: AmazonS3
|
||||
```
|
||||
|
||||
…with the response body:
|
||||
|
||||
```xml
|
||||
<Error>
|
||||
<Code>AccessDenied</Code>
|
||||
<Message>Access Denied</Message>
|
||||
...
|
||||
</Error>
|
||||
```
|
||||
|
||||
The hostname is the remote instance's S3 bucket (e.g. `s3.amazonaws.com/<their-bucket>/accounts/avatars/...`). Other actors in the same notifications list — those with avatars on Mastodon's own CDN, or on instances using path-stable storage — render fine.
|
||||
|
||||
This article explains *why* the alarm code is misleading, *what's actually broken*, and how to fix it on Castopod.
|
||||
|
||||
---
|
||||
|
||||
## 🔬 Why "AccessDenied" is misleading
|
||||
|
||||
S3 returns `403 AccessDenied` to anonymous requesters for **any** missing object — by design, as anti-enumeration. Anonymous users typically don't have `s3:ListBucket` permission on the bucket, so S3 deliberately can't tell them whether the key is missing or merely forbidden. Both cases produce the same 403.
|
||||
|
||||
So when you see `403 AccessDenied` on a remote avatar URL, **the actual problem is almost always that the object no longer exists**. The bucket is fine; the file is gone.
|
||||
|
||||
### Verifying that interpretation
|
||||
|
||||
If you have access to the remote instance (or to S3 credentials for that bucket):
|
||||
|
||||
```sh
|
||||
aws s3api head-object --bucket <bucket> --key accounts/avatars/.../<filename>.jpeg
|
||||
```
|
||||
|
||||
If you see `An error occurred (404) when calling the HeadObject operation: Not Found`, the object is genuinely gone — and the upstream user has updated their avatar.
|
||||
|
||||
---
|
||||
|
||||
## 🔍 What's actually broken
|
||||
|
||||
Mastodon (and most ActivityPub servers using Paperclip-style storage) **deletes the old object** on avatar replacement and stores only the current filename in the DB. The remote instance is functioning normally — its current `<img>` URL points to a different filename and serves correctly.
|
||||
|
||||
Castopod 2.0.0 (verified up to `2.0.0-next.4`) **caches the avatar URL** of every federated actor in `cp_fediverse_actors.avatar_image_url` when it first sees activity from that actor — and never refetches. The admin templates (e.g. `themes/cp_admin/podcast/notifications.php`) emit that stored URL directly into `<img src>`. Once the upstream replaces the avatar:
|
||||
|
||||
- Old object deleted → S3 returns 403 to anonymous fetchers
|
||||
- Castopod still renders the dead URL forever
|
||||
- Every cached page using that template shows a broken image
|
||||
|
||||
The same pattern applies to `cover_image_url` (header).
|
||||
|
||||
---
|
||||
|
||||
## ✅ Fix
|
||||
|
||||
You have three options, in increasing order of "this stays fixed."
|
||||
|
||||
### Option 1 — Manual SQL update (one-shot)
|
||||
|
||||
Recommended for one or two stale actors. Get the current URL from the upstream instance.
|
||||
|
||||
If the upstream is your own Mastodon instance:
|
||||
|
||||
```sh
|
||||
sudo -u postgres psql mastodon_production -t -A \
|
||||
-c "SELECT id, avatar_file_name, header_file_name FROM accounts WHERE username='<their-username>'"
|
||||
```
|
||||
|
||||
Construct the canonical URL using the standard Paperclip path scheme. For an account ID like `109326168175475699`, the path is built by chunking the ID three digits at a time:
|
||||
|
||||
```
|
||||
accounts/avatars/109/326/168/175/475/699/original/<avatar_file_name>
|
||||
accounts/headers/109/326/168/175/475/699/original/<header_file_name>
|
||||
```
|
||||
|
||||
Then UPDATE the Castopod row:
|
||||
|
||||
```sh
|
||||
mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME <<'SQL'
|
||||
UPDATE cp_fediverse_actors
|
||||
SET avatar_image_url = 'https://<s3-host>/<bucket>/accounts/avatars/109/326/168/175/475/699/original/<new>.jpeg',
|
||||
cover_image_url = 'https://<s3-host>/<bucket>/accounts/headers/109/326/168/175/475/699/original/<new>.jpg',
|
||||
updated_at = NOW()
|
||||
WHERE username = '<their-username>'
|
||||
AND domain = '<their-domain>';
|
||||
SQL
|
||||
```
|
||||
|
||||
Then clear the Castopod cache so any cached HTML rerenders:
|
||||
|
||||
```sh
|
||||
cd /var/www/html/castopod
|
||||
sudo -u www-data php spark cache:clear
|
||||
```
|
||||
|
||||
Verify:
|
||||
|
||||
```sh
|
||||
curl -sI 'https://<new-url>' | head -1 # expect HTTP/1.1 200 OK
|
||||
```
|
||||
|
||||
### Option 2 — Delete and let Castopod refetch
|
||||
|
||||
For a one-shot self-healing fix, delete the actor row entirely:
|
||||
|
||||
```sql
|
||||
DELETE FROM cp_fediverse_actors WHERE username='<u>' AND domain='<d>';
|
||||
```
|
||||
|
||||
Castopod will repopulate the row from the next inbound activity from that actor (favourite, boost, mention, follow…). **Caveat — verify foreign-key cascades first:** `cp_fediverse_favourites`, `cp_fediverse_follows`, `cp_fediverse_posts`, and `cp_fediverse_notifications` all reference `actor_id`. Depending on the migration version, ON DELETE may cascade or restrict. Check with:
|
||||
|
||||
```sh
|
||||
mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME -e "
|
||||
SELECT TABLE_NAME, CONSTRAINT_NAME, DELETE_RULE
|
||||
FROM information_schema.REFERENTIAL_CONSTRAINTS
|
||||
WHERE CONSTRAINT_SCHEMA = '$CP_DB_NAME'
|
||||
AND REFERENCED_TABLE_NAME = 'cp_fediverse_actors';
|
||||
"
|
||||
```
|
||||
|
||||
If deletes cascade, you'll lose the activity history attributed to that actor. Use Option 1 instead.
|
||||
|
||||
### Option 3 — Bulk audit and update
|
||||
|
||||
If multiple federated actors have likely-stale avatars (any old enough that an upstream user might have refreshed their profile picture), audit them all:
|
||||
|
||||
```sh
|
||||
mysql -u $CP_DB_USER -p$CP_DB_PASS $CP_DB_NAME -BNe "
|
||||
SELECT id, username, domain, avatar_image_url
|
||||
FROM cp_fediverse_actors
|
||||
WHERE avatar_image_url IS NOT NULL
|
||||
" | while IFS=$'\t' read -r id user dom url; do
|
||||
code=$(curl -s -o /dev/null -w "%{http_code}" "$url")
|
||||
[ "$code" != "200" ] && echo "BROKEN $code $id $user@$dom $url"
|
||||
done
|
||||
```
|
||||
|
||||
For each broken row, fetch the upstream's current actor JSON and update from `icon.url` / `image.url`:
|
||||
|
||||
```sh
|
||||
curl -s -H 'Accept: application/activity+json' \
|
||||
"https://<their-domain>/users/<their-username>" | jq '{icon, image}'
|
||||
```
|
||||
|
||||
Then run the Option 1 SQL update with the fresh URLs.
|
||||
|
||||
---
|
||||
|
||||
## 🧪 Why this isn't fixable on the upstream side
|
||||
|
||||
Once the old object is deleted, you can't restore the URL without re-uploading bytes to the **exact original key** — which Mastodon won't do, because its DB only knows about the new filename. Trying to "fix" it on the Mastodon side means resurrecting a file Mastodon has no record of and that no fresh ActivityPub request would emit a URL for. The fix has to live on the consumer (Castopod) because Castopod is the one holding the stale reference.
|
||||
|
||||
This applies to every federation consumer that caches URLs by reference rather than fetching bytes locally. Mastodon, Pleroma, Akkoma, and Misskey all cache the bytes; that's why they self-heal across remote avatar swaps. Castopod 2.0.0 currently does not.
|
||||
|
||||
---
|
||||
|
||||
## 🛠 Long-term mitigations
|
||||
|
||||
This is a Castopod design issue worth raising upstream:
|
||||
- Add a `last_refreshed_at` to `cp_fediverse_actors` and a worker that refetches actor JSON on a schedule.
|
||||
- Or fetch and store avatars locally on first sight, the way Mastodon does.
|
||||
|
||||
A `fediverse:refresh-actor` spark command would also let admins fix stale rows without writing SQL.
|
||||
|
||||
If you have a recurring case (you update your Mastodon avatar often, and you also operate a Castopod instance under your own control), keep the Option 1 SQL handy as a one-liner. After your own avatar update, run it within minutes and the dead-URL window closes before it spreads to many cached pages.
|
||||
|
||||
---
|
||||
|
||||
## 📚 References
|
||||
|
||||
- Castopod source (`themes/cp_admin/podcast/notifications.php`) — uses `avatar_image_url` directly in `<img src>`
|
||||
- AWS S3 anti-enumeration: `403` vs `404` is bucket-policy-dependent; see [GetObject — Permissions Required](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html#API_GetObject_RequestPermissions)
|
||||
- Mastodon Paperclip storage layout: `accounts/avatars/<3-digit chunks of account id>/original/<file_name>`
|
||||
- Related fix patterns: [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](netdata-web-log-successful-redirect-heavy-tuning.md) — shares the "the alarm is technically correct, but means something different than you think" theme
|
||||
|
|
@ -76,6 +76,7 @@ updated: 2026-05-08T01:08
|
|||
* [Fail2ban & UFW Rule Bloat Cleanup](05-troubleshooting/networking/fail2ban-ufw-rule-bloat-cleanup.md)
|
||||
* [Custom Fail2ban Jail: Apache Directory Scanning](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md)
|
||||
* [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md)
|
||||
* [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md)
|
||||
* [Nextcloud AIO Unhealthy 20h After Nightly Update](05-troubleshooting/docker/nextcloud-aio-unhealthy-20h-stuck.md)
|
||||
* [n8n Behind Reverse Proxy: X-Forwarded-For Trust Fix](05-troubleshooting/docker/n8n-proxy-trust-x-forwarded-for.md)
|
||||
* [Docker & Caddy Recovery After Reboot (Fedora + SELinux)](05-troubleshooting/docker-caddy-selinux-post-reboot-recovery.md)
|
||||
|
|
|
|||
6
index.md
6
index.md
|
|
@ -7,7 +7,7 @@ updated: 2026-05-08T01:08
|
|||
> A growing reference of Linux, self-hosting, open source, streaming, and troubleshooting guides. Written by MajorLinux. Used by MajorTwin.
|
||||
>
|
||||
> **Last updated:** 2026-05-08
|
||||
> **Article count:** 107
|
||||
> **Article count:** 108
|
||||
|
||||
## Domains
|
||||
|
||||
|
|
@ -17,7 +17,7 @@ updated: 2026-05-08T01:08
|
|||
| 🏠 Self-Hosting & Homelab | `02-selfhosting/` | 39 |
|
||||
| 🔓 Open Source Tools | `03-opensource/` | 10 |
|
||||
| 🎙️ Streaming & Podcasting | `04-streaming/` | 2 |
|
||||
| 🔧 General Troubleshooting | `05-troubleshooting/` | 44 |
|
||||
| 🔧 General Troubleshooting | `05-troubleshooting/` | 45 |
|
||||
|
||||
|
||||
---
|
||||
|
|
@ -201,6 +201,7 @@ updated: 2026-05-08T01:08
|
|||
- [ClamAV Safe Scheduling on Live Servers](05-troubleshooting/security/clamscan-cpu-spike-nice-ionice.md)
|
||||
- [Custom Fail2ban Jail: Apache Directory Scanning & Junk Methods](05-troubleshooting/security/apache-dirscan-fail2ban-jail.md)
|
||||
- [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md)
|
||||
- [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md)
|
||||
|
||||
### Storage
|
||||
- [mdadm RAID Recovery After USB Hub Disconnect](05-troubleshooting/storage/mdadm-usb-hub-disconnect-recovery.md)
|
||||
|
|
@ -215,6 +216,7 @@ updated: 2026-05-08T01:08
|
|||
|
||||
| Date | Article | Domain |
|
||||
|---|---|---|
|
||||
| 2026-05-08 | [Castopod: Stale Federated Avatar URLs After Remote Profile Updates](05-troubleshooting/security/castopod-stale-federated-avatar.md) | Troubleshooting |
|
||||
| 2026-05-08 | [Tuning Netdata `web_log_1m_successful` for Redirect-Heavy WordPress Sites](05-troubleshooting/security/netdata-web-log-successful-redirect-heavy-tuning.md) | Troubleshooting |
|
||||
| 2026-05-07 | [Mastodon — The `--prune-profiles` Trap and How to Recover](02-selfhosting/services/mastodon-prune-profiles-trap.md) | Self-Hosting |
|
||||
| 2026-05-02 | [WSL2 Backup via PowerShell Scheduled Task](01-linux/distro-specific/wsl2-backup-powershell.md) | Linux |
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue