CDN Distribution and Cache Invalidation

Within the Media Delivery & Publishing Automation layer, CDN distribution is the stage that moves finished artifacts — packaged video renditions, podcast enclosures, playlists, and the RSS feed — into object storage that sits behind an edge cache, and then keeps that edge honest as episodes change. Get the cache policy right and a single origin serves millions of listener requests at a near-100% offload; get it wrong and you either hammer origin on every segment or, worse, keep serving a stale enclosure to half your audience for hours after you fixed it. This stage treats caching as a contract: each asset class declares exactly how long an edge may hold it, how it is grouped for invalidation, and whether a byte-for-byte overwrite is even permitted.

The unit of correctness here is the pair (cache policy, purge strategy). Immutable, content-hashed assets are cached forever and never purged because their URL changes when their bytes change. Mutable, stable-URL assets — the playlist that points at new segments, the feed that lists a re-tagged episode — carry short edge TTLs and are actively invalidated the moment they change. Everything below builds that split and automates it.

Prerequisites & Environment

Pin every dependency. Purge semantics and header handling differ subtly across CDN and SDK versions, and a silent behavior change in caching is exactly the kind of bug that only surfaces in a listener’s podcast app three time zones away.

• Python: 3.10+ (examples use match statements and tomllib).
• CDN: a Fastly service fronting your object-storage origin, with an API token scoped to purge and purge_all. This page uses Fastly throughout because its surrogate-key purging model maps cleanly onto per-episode invalidation; the same shape (tag on write, purge by tag) applies to any tag-capable CDN.
• Object storage: any S3-compatible bucket used as the Fastly origin.
• Python libraries (pinned):

# fastly==13.1.0          # official Fastly API client; PurgeApi lives here
# boto3==1.34.84          # S3-compatible uploads with per-object Cache-Control
# httpx==0.27.0           # header verification against the edge
# tenacity==8.2.3         # rate-limit-aware retry/backoff around purge calls

• Environment variables the distributor reads at startup:

FASTLY_API_TOKEN=fastly_pat_xxx        # scoped to purge; never a full-admin token
FASTLY_SERVICE_ID=SU1Z0isxPaozGVKXdv0eY
CDN_BASE_URL=https://cdn.example.com
ORIGIN_BUCKET=media-origin-prod
SEGMENT_MAX_AGE=31536000               # 1 year, for immutable content-hashed assets
PLAYLIST_MAX_AGE=6                      # seconds of edge freshness for HLS/DASH manifests
FEED_MAX_AGE=300                        # seconds of edge freshness for the RSS feed
PURGE_SOFT=1                            # mark stale instead of hard-evicting (stampede-safe)

The three TTL knobs are the whole policy in miniature: a year for hashed media, single-digit seconds for manifests, a few minutes for the feed. Bake them into configuration so a delivery-spec change never requires a code deploy.

Architecture & Queue Topology

A publish event fans into a short, ordered pipeline: an upload worker writes the new objects to origin, cache headers are stamped per asset class, and only after the write is durable does a targeted purge invalidate the mutable objects at the edge — followed by a warm-and-verify pass that proves propagation before the release is announced. Ordering is not optional: purge before the origin write is durable and the edge will re-cache the old object on the next request.

The two lower boxes are where all the nuance lives. The origin holds both immutable media (whose keys embed a content hash) and the mutable feed (whose URL is stable). The edge serves both, but only the mutable objects are ever the target of a purge — and that asymmetry is what keeps invalidation cheap. This distribution stage consumes the exact artifacts produced by adaptive bitrate packaging and the XML emitted by podcast RSS feed automation; it never re-encodes or re-serializes, it only places and invalidates.

Step-by-Step Implementation

Each step ends with a verification command you can run before promoting the release.

1. Compute content-hashed object keys

Media segments, init segments, enclosures, and cover art all get a key that embeds a short content hash. When the bytes change, the key changes, so a cached copy at any tier is never wrong — it is simply a different URL. Manifests and the feed keep stable keys because clients discover them by a fixed address.

# python 3.10+
import hashlib, pathlib

def hashed_key(local_path: str, prefix: str, ext: str) -> str:
    """content-addressed key: media/<prefix>/<sha1[:12]>.<ext>"""
    digest = hashlib.sha1(pathlib.Path(local_path).read_bytes()).hexdigest()[:12]
    return f"media/{prefix}/{digest}.{ext}"

def stable_key(prefix: str, name: str) -> str:
    """stable key for objects discovered by a fixed URL (manifests, feed)."""
    return f"{prefix}/{name}"

Verify that identical bytes map to identical keys and a single flipped byte does not:

python -c "import dist; print(dist.hashed_key('seg0.m4s','ep-1041','m4s'))"
# media/ep-1041/9f2a1c4b7e03.m4s  -> stable across re-runs of the same bytes

2. Upload with class-specific cache headers

The upload worker stamps Cache-Control at PUT time and attaches a Surrogate-Key (space-separated tags) plus a Surrogate-Control edge TTL as origin metadata that Fastly reads on fill. Immutable assets get a one-year max-age with the immutable token so browsers never even send a revalidation request; manifests and the feed get single-digit-second to few-minute edge freshness; the feed alone is compressed.

# boto3==1.34.84
import boto3, os

s3 = boto3.client("s3", endpoint_url=os.environ.get("S3_ENDPOINT"))
BUCKET = os.environ["ORIGIN_BUCKET"]
SEG = os.environ["SEGMENT_MAX_AGE"]
FEED = os.environ["FEED_MAX_AGE"]

def put_segment(key: str, body: bytes, episode: str) -> None:
    s3.put_object(
        Bucket=BUCKET, Key=key, Body=body,
        ContentType="video/iso.segment",
        CacheControl=f"public, max-age={SEG}, immutable",
        Metadata={"surrogate-key": f"ep-{episode} segment",
                  "surrogate-control": f"max-age={SEG}"},
    )

def put_feed(key: str, xml_gz: bytes, show: str) -> None:
    s3.put_object(
        Bucket=BUCKET, Key=key, Body=xml_gz,
        ContentType="application/rss+xml; charset=utf-8",
        ContentEncoding="gzip",                       # feed only — never media
        CacheControl=f"public, max-age={FEED}, must-revalidate",
        Metadata={"surrogate-key": f"show-{show} feed",
                  "surrogate-control": f"max-age={FEED}"},
    )

Compress the feed with gzip or brotli because XML is highly compressible and shrinks 80%+ on the wire; never apply content-coding to already-compressed segments or audio enclosures — it wastes CPU and, critically, can break byte-Range requests. Enclosures must be served with Accept-Ranges: bytes so podcast players can seek and resume partial downloads; confirm your origin and Fastly service pass Range through rather than collapsing to a full 200.

Verify the header landed on origin and, after a fill, at the edge:

curl -sI "$CDN_BASE_URL/media/ep-1041/9f2a1c4b7e03.m4s" | grep -i 'cache-control\|surrogate'
# cache-control: public, max-age=31536000, immutable

3. Purge the mutable objects by surrogate key

Only the manifest and the feed change identity-in-place, so those are the purge targets. Group each episode’s mutable objects under a shared key (ep-<id>) so one call invalidates the playlist, the master manifest, and any thumbnail sprite atomically. Use soft purge (Fastly-Soft-Purge: 1) so the edge serves the stale object once while it revalidates, which prevents an origin stampede on popular shows.

# fastly==13.1.0
import os, fastly
from fastly.api import purge_api

def _purge_api() -> purge_api.PurgeApi:
    cfg = fastly.Configuration()
    cfg.api_token = os.environ["FASTLY_API_TOKEN"]
    return purge_api.PurgeApi(fastly.ApiClient(cfg))

def purge_episode(episode: str, soft: bool = True) -> dict:
    api = _purge_api()
    return api.purge_tag(
        service_id=os.environ["FASTLY_SERVICE_ID"],
        surrogate_key=f"ep-{episode}",
        fastly_soft_purge=1 if soft else None,
    ).to_dict()

Verify the purge returned an id and that a subsequent request is a fresh copy:

python -c "import dist; print(dist.purge_episode('1041'))"
# {'status': 'ok', 'id': '...'}   -> a purge id confirms acceptance

4. Warm and verify propagation

A purge is asynchronous across POPs. Before you announce a release, prefetch the manifest and the feed from the edge and assert the new content is being served, so you never advertise a URL that half the network still answers from stale cache.

# httpx==0.27.0
import httpx, os

def warm_and_check(paths: list[str], expect_etag: dict[str, str]) -> bool:
    base = os.environ["CDN_BASE_URL"]
    with httpx.Client(timeout=10) as c:
        for p in paths:
            r = c.get(f"{base}/{p}", headers={"Fastly-Debug": "1"})
            r.raise_for_status()
            if expect_etag.get(p) and r.headers.get("etag") != expect_etag[p]:
                return False
    return True

Verify at the shell that the edge reports a hit on the warmed object and a matching Age:

curl -sI "$CDN_BASE_URL/show-hn/feed.xml" | grep -i 'x-cache\|age\|etag'
# x-cache: HIT, HIT   age: 4   etag: "b91c…"  -> fresh feed, freshly warmed

The narrower republish case — re-encoding or re-tagging an already-published episode and defeating a same-URL stale copy — is covered end to end in invalidating CDN cache on episode republish.

Data Contracts

Every asset that leaves this stage carries a declared cache policy. The table is the contract the upload worker enforces and the purge worker relies on; treat any object without a matching row as a defect.

The single most important row is the one you do not purge: content-hashed media. Because its identity is a function of its bytes, a change produces a new URL and the stale copy is simply never requested again — invalidation cost is zero, at any scale.

Resilience Patterns

Purge is a networked side effect on a rate-limited API, so treat it with the same discipline as any other external call.

• Write-before-purge ordering. Confirm the origin PUT is durable (a successful put_object with the expected ETag) before issuing the purge. Purging first lets a concurrent request refill the edge from the old origin object.
• Soft over hard purge. Soft purge marks objects stale rather than evicting them, so the first post-purge request is served instantly from stale cache while revalidation happens in the background — no origin stampede, no user-visible latency spike.
• Rate-limit budgeting. Surrogate-key and URL purges are limited to an average of 100,000 purges per hour per customer and are not counted against general API rate limits; batch related keys with bulk_purge_tag (up to 256 keys per request) rather than looping single-key calls. Wrap purges in tenacity with exponential backoff and jitter, and treat HTTP 429 as retryable.
• Purge-all as break-glass only. A purge_all cannot be soft and immediately invalidates the whole service, collapsing your hit ratio to zero and stampeding origin. Reserve it for a poisoned-config incident, never for routine republishes.
• Idempotent purges. A purge is naturally idempotent — re-issuing ep-1041 after a transient failure is safe. Key the purge job on (episode_id, content_etag) so a retried worker does not double-count metrics or fire redundant warm requests.
• Failure routing. A purge that exhausts retries is a delivery-blocking event: route it to the shared retry logic and dead-letter queues so a stuck edge is visible and replayable rather than silently stale.

Observability & Debugging

Instrument the distributor so a stale edge is a metric, not a support ticket.

• Key metrics (Prometheus):
  • cdn_purge_requests_total — counter, labeled by key_type (segment should always be zero) and result (ok, rate_limited, error).
  • cdn_purge_latency_seconds — histogram from purge acceptance to a verified fresh edge response.
  • cdn_cache_hit_ratio — gauge scraped from Fastly real-time stats; a sustained drop signals a bad TTL or an over-broad purge.
  • cdn_stale_served_total — counter incremented when a verify probe finds an unexpected old ETag after purge.
  • origin_egress_bytes_total — counter; a spike after deploy usually means an immutable asset lost its long max-age.
• Structured log fields: episode_id, object_key, surrogate_key, cache_control, purge_id, soft, edge_etag, x_cache, age_s, attempt.
• Common errors and root causes:
  • Listeners get the old file after a fix — a stable-URL asset was overwritten but not purged, or the purge ran before the origin write was durable. Re-order and re-purge.
  • Hit ratio collapsed after deploy — a manifest or feed inherited the immutable one-year policy, or a purge_all fired; audit the header per asset class.
  • Player cannot seek in an enclosure — origin dropped Accept-Ranges or a content-coding was applied to the audio; disable compression for media and confirm Range passthrough.
  • Fastly-Debug shows PASS not HIT — an unexpected Set-Cookie or Cache-Control: private on the object is defeating caching entirely.

Performance Tuning

• Maximize immutable share. The lever with the highest return is pushing as many bytes as possible into content-hashed, immutable URLs. Every immutable byte is cached forever at every tier and never purged, so segments and cover art should never carry a short TTL.
• Enable origin shield. A single shield POP collapses global misses into one origin request; without it, a cold object can trigger one origin fetch per POP. Pair it with request collapsing so a thundering herd on a fresh manifest is de-duplicated at the shield.
• Keep manifest TTL as high as tolerable. For live-adjacent workflows the manifest TTL bounds seek-to-live latency, but for on-demand podcasts a 6-second edge TTL already offloads virtually all manifest traffic; do not set it to zero out of caution.
• Compress the feed, only the feed. Brotli on the RSS XML cuts feed egress dramatically at negligible CPU; leave media and audio uncompressed so Range and byte-accurate seeking keep working.
• Batch purges. Collapse per-object purges into a single bulk_purge_tag per release to stay well under the hourly purge budget and to make invalidation atomic from a listener’s perspective.

Frequently Asked Questions

Related

• Media Delivery & Publishing Automation — the parent layer this distribution stage belongs to.
• Adaptive Bitrate Packaging — produces the content-hashed segments and manifests this stage places behind the edge.
• Podcast RSS Feed Automation — emits the short-TTL feed that is the primary purge target here.
• Invalidating CDN Cache on Episode Republish — the focused fix for a stale same-URL enclosure after a re-encode.
• Retry logic and dead-letter queues — where a purge that exhausts retries is routed for replay.