majorwiki/02-selfhosting/cloud/aws-s3-cost-management.md

title	domain	category	tags	status	created	updated
AWS S3 Cost Management	selfhosting	cloud	aws s3 cost billing mastodon glacier	published	2026-04-19	2026-06-01

AWS S3 Cost Management

The majorlinux AWS account is used exclusively for S3 object storage. This covers how to monitor costs, what's driving the bill, and how to reduce it.

Account Overview

• Account ID: 408469496267
• Account name: majorlinux
• Services in use: S3 (Standard + Glacier Deep Archive), AWS Config, Cost Explorer
• Monthly spend: ~$24/mo (May 2026, post-media-prune, post-STANDARD_IA revert)

Buckets and Cost Drivers

Bucket	Size	Storage Class	Cost/mo	Purpose
majortoot	~7 GB (one-time prune; automation disabled)	S3 Standard	~$0.16/mo	Mastodon media
majorhomebackup	16 TiB	Glacier Deep Archive	~$11–12/mo	MLS stream archives (sole copy)
config-bucket-*	~185 KB	S3 Standard	~$0.00	AWS Config snapshots

CLI Setup

AWS CLI installed on MajorMac via Homebrew. Credentials for MajorCLI user at ~/.aws/credentials.

brew install awscli
# Credentials: MajorCLI IAM user (S3 + Billing read access)
# Key ID: AKIAV6GVN4HF4Y6EV4NM — created 2026-05-23

Useful commands

# Check current month spend by service
aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-05-31 \
  --granularity MONTHLY \
  --metrics "UnblendedCost" \
  --group-by Type=DIMENSION,Key=SERVICE

# Daily cost breakdown with top usage types
aws ce get-cost-and-usage \
  --time-period Start=2026-05-01,End=2026-05-23 \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --filter '{"Dimensions":{"Key":"SERVICE","Values":["Amazon Simple Storage Service"]}}' \
  --group-by Type=DIMENSION,Key=USAGE_TYPE

# View anomaly alerts
aws ce get-anomalies \
  --date-interval StartDate=2026-05-01,EndDate=2026-05-31

# List budgets
aws budgets describe-budgets --account-id 408469496267

Budget Alert

MajorS3MonthlyAlert configured 2026-04-19:

• 80% threshold → email at $24 actual spend
• 100% threshold → email at $30 actual spend
• Recipient: maj.linux@gmail.com

[!note] Thresholds updated 2026-05-23 to reflect actual ~$24/mo steady-state spend (was $20/$25, set when spend was higher due to large majortoot bucket before prune took effect).

Cost Reduction Options

majortoot — S3 Standard-IA (⚠️ DO NOT USE — tried and reverted)

Attempted 2026-05 — reverted 2026-05-17. Do not retry without careful planning.

The theory: switching S3_STORAGE_CLASS=STANDARD_IA saves ~$4–5/mo on storage. In practice, the bulk avatar restore operation (restore-avatars.sh, May 9–10) ran while STANDARD_IA was active. The ~5,223 account refreshes across 1,095 domains generated ~470,000 SIA Tier 1 PUT requests ($4.72) plus early-deletion fees ($1.21) when the objects were replaced after reverting to STANDARD on May 17.

STANDARD_IA is only economical if:

• The bucket has no large bulk-write operations (media cache rebuilds, avatar restores)
• Objects are written and left for >30 days (early deletion incurs minimum 30-day fee)
• The per-request cost ($0.01/1,000 for SIA vs $0.005/1,000 for Standard) doesn't offset storage savings

With the weekly prune now running correctly and the bucket shrinking toward 7 GB, the storage savings of SIA are negligible ($0.05/mo). Leave at STANDARD.

majortoot — media pruning (automation DISABLED 2026-06-01)

A weekly prune cron (0 3 * * 0, via configure_mastodon_media_prune.yml) used to run tootctl media remove --days=7. It shrank the bucket from 648 GB to ~7 GB — a one-time cleanup of years of accumulated remote attachment cache, which is safe and accounts for the bulk of the savings above.

That automation was removed 2026-06-01. The same playbook also carried a monthly tootctl accounts refresh --all, and automated profile pruning (plus a storage-level deletion during the cost-cull/migration) repeatedly broke remote avatars. The playbook is now an enforce-absent guard, and a synthetic upload health check alerts if media serving/uploads regress. See mastodon-prune-profiles-trap and mastodon-s3-acl-upload-failures.

Going forward: the bucket is already small (~7 GB) and attachment cache re-accumulates slowly. If it ever grows enough to matter, run an attachment-only prune manually and deliberately (bin/tootctl media remove --days=30) — never automate profile/header pruning or accounts refresh --all.

majorhomebackup — Self-host consideration

Deep Archive at $0.00099/GB is the cheapest cloud tier — no cloud alternative is cheaper. If the MLS archives are no longer needed, deletion would save $11–12/mo. A 20TB HDD ($300–400) would break even in ~2.5 years vs. continued cloud storage. These are the sole copy — do not delete without a separate backup.

IAM Users

User	Scope	Credentials location	Notes
MajorToot	S3 full (MajorsHouse group)	~/.aws/credentials on majortoot	Key rotated 2026-05-23
MajorHome	S3 full (MajorsHouse group)	~/.aws/credentials on majorhome	Key pending rotation (see below)
MajorCLI	S3 full + Billing read (MajorsHouse group + AWSBillingReadOnlyAccess)	~/.aws/credentials on MajorMac	Created 2026-05-23, replaces root key

[!warning] Root access keys deleted 2026-05-23. Do NOT create new root access keys. Use MajorCLI for CLI work on MajorMac. The root account password (in Vaultwarden) is sufficient for console access.

[!warning] MajorHome key (AKIAV6GVN4HF7POCNW6D) exposed in shell session 2026-05-23. Rotate via AWS Console → IAM → Users → MajorHome → Security credentials. Update ~/.aws/credentials on majorhome afterward.

[!note] MajorCLI does not have IAM permissions. Future key rotation requires AWS Console login or temporary IAM policy attachment. Consider adding a SelfManageKeys inline policy to MajorCLI via console.

Conformance Pack

MajorConformance (created 2024-12-20) monitors S3 buckets for:

• Public read/write access (majortoot is intentionally public — Mastodon media)
• Account-level public access blocks (off by design, same reason)
• S3 default object lock (not enabled — expected)
• S3 event notifications (not enabled — expected)

Evaluations cost $0.001 each and run on a periodic schedule. Safe to ignore; at current scale costs pennies per month.

CloudTrail Audit Logging

MajorTrail configured 2026-05-23:

• S3 bucket: majorcloudtrail-408469496267
• Multi-region: yes — captures API calls across all regions
• Global service events: yes — includes IAM, STS, S3 control plane
• Log file validation: enabled — tamper detection via digest files
• Retention: logs accumulate in S3; no automatic expiry configured

Use CloudTrail to investigate unexpected cost spikes, IAM key usage, and bucket write activity. Without it, historical API calls are unrecoverable (learned the hard way from the May 2026 SIA spike investigation).

# List recent CloudTrail events (last 1h, S3 writes only)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=PutObject \
  --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%SZ 2>/dev/null || date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --query 'Events[].{Time:EventTime,User:Username,Resource:Resources[0].ResourceName}' \
  --output table

# Look up events by specific access key
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAV6GVN4HF3BWAIAGC \
  --output table

Related

• mastodon-instance-tuning — media cache management
• mastodon-prune-profiles-trap — avatar restore incident + bulk-restore procedure
• mastodon-s3-acl-upload-failures — silent upload failures on ACL-disabled buckets
• majortoot — Mastodon host