Media Delivery & Publishing Automation

Delivery is the last stage of an automated podcast and video pipeline, and it is the one most likely to embarrass a team in public. Everything upstream — ingestion, transcoding, transcription, chaptering — happens in private, where a failure is a stalled worker or a quarantined asset. Delivery is where the pipeline touches the outside world: it packages a validated, transcribed, chaptered asset into the exact renditions a player expects, wraps them in a podcast feed that Apple and Spotify will actually accept, and pushes bytes onto a content delivery network with caching rules that make the difference between a listener hearing the new episode and hearing yesterday’s cached one. Without a deterministic delivery layer, teams ship feeds that fail platform validation, serve stale episodes from CDN edges long after a republish, break chapter markers that were correct the moment they left the encoder, and — most insidiously — mismatch the byte length declared in an RSS <enclosure> against the object actually stored on the CDN, which silently corrupts progress bars and range requests across every podcast client. This layer exists to make the published artifact a deterministic function of its validated inputs, so that publishing is a mechanical promotion rather than a nightly gamble. It consumes the outputs of the Media Ingestion & Format Architecture, Transcription & Speaker Diarization, and Pipeline Automation & Batch Processing layers and turns them into something a distribution platform will fetch.

Architecture Overview

Delivery is a linear promotion pipeline with four gates, each of which transforms the asset and each of which prevents a specific class of public-facing failure. A normalized, transcribed asset — a faststart MP4 or PCM master, a WebVTT transcript, and a chapter cue list — enters the first gate and is packaged into an adaptive-bitrate rendition ladder. The second gate embeds chapters, artwork, and metadata into the deliverable containers so that the markers survive transport. The third gate generates the podcast RSS feed and validates it against the platform schemas before anyone can subscribe to a broken document. The fourth gate uploads the segments and feed to the CDN and invalidates exactly the cache keys that changed. Only after all four gates clear does an episode become “published” in the sense that matters: cache-warm at the edge and passing Apple and Spotify validation. Nothing is published half-way — a feed is never made live until its enclosures resolve, and a cache is never left holding a stale manifest.

Each gate maps to a documented sub-stage of this architecture: rendition-ladder generation via Adaptive Bitrate Packaging, marker and tag durability via Chapter and Metadata Embedding, feed construction and conformance via Podcast RSS Feed Automation, and edge distribution via CDN Distribution and Cache Invalidation. The sections below cover each gate’s purpose, a concrete implementation sketch, and the specific failure mode it exists to prevent.

Core Data Contracts & Normalization Rules

The delivery layer enforces its own contracts, distinct from ingestion’s. Where ingestion guarantees a clean master, delivery guarantees a valid deliverable: a container, codec, segment layout, and feed field set that a specific consumer — an HLS player, an ID3-aware podcast app, or the Apple Podcasts validator — will accept without complaint. A deliverable that violates its contract is never published; it is failed back with a deterministic error code, exactly as ingestion quarantines a bad master. The table below is the canonical deliverable contract set this architecture enforces. Every gate validates against the relevant columns and emits the listed error code on a miss.

Three normalization rules govern every promotion. First, addressing rules: every URL that leaves this layer — enclosure, manifest, artwork, chapter JSON — is an absolute, HTTPS, CDN-fronted URL, never a relative path or an origin URL that would bypass the edge. Second, length rules: the byte length declared for any enclosure is computed from the exact object that will be served, after packaging and any container mutation, and re-verified against the CDN’s Content-Length before the feed goes live; a declared length that disagrees with the served object by a single byte breaks seeking in strict clients. Third, immutability rules: a published segment or enclosure object is content-addressed or versioned so that republishing an episode never mutates a URL a cache is already holding — new bytes get a new key, and the feed points at the new key. A contract miss is never coerced past tolerance; it is serialized as a structured error and failed back to the Pipeline Automation & Batch Processing orchestrator that scheduled the publish.

Adaptive Bitrate Packaging

The first gate turns a single normalized master into a ladder of renditions and the manifest that ties them together. Adaptive streaming exists because networks are not uniform: a listener on a train needs the player to drop to a 64 kbps audio rendition or a 360p video rendition without a rebuffer, and to climb back up when bandwidth returns. That only works if the master is segmented on aligned boundaries across every rendition, so that a player can switch variants at any segment edge without a discontinuity. Implementing Adaptive Bitrate Packaging means encoding the ladder from the faststart master handed over by ingestion, cutting every rendition on identical keyframe-aligned segment boundaries, and emitting an HLS master playlist or DASH MPD that advertises each variant’s bandwidth accurately. The packaging step is where CMAF fragmented-MP4 output pays off: one set of segments can back both an HLS and a DASH manifest, halving storage and cache footprint.

# python 3.10+
# ffmpeg 6.1 (libx264); Bento4 1.6.0 (mp4hls) for CMAF packaging
import subprocess
from pathlib import Path

LADDER = [  # (height, video_kbps, audio_kbps)
    (360, 800, 96),
    (720, 2400, 128),
    (1080, 4800, 128),
]
SEG_SECONDS = 6  # keyframe interval must equal this so variants align

def package_hls(master: Path, out_dir: Path) -> Path:
    """Encode a keyframe-aligned ladder and emit an HLS master playlist."""
    args = ["ffmpeg", "-hide_banner", "-y", "-i", str(master)]
    var_map, stream_map = [], []
    for i, (h, vk, ak) in enumerate(LADDER):
        gop = SEG_SECONDS * 30  # 30 fps source -> fixed GOP == segment length
        args += [
            "-map", "0:v:0", "-map", "0:a:0",
            f"-c:v:{i}", "libx264", f"-b:v:{i}", f"{vk}k",
            f"-vf", f"scale=-2:{h}",
            "-g", str(gop), "-keyint_min", str(gop), "-sc_threshold", "0",
            f"-c:a:{i}", "aac", f"-b:a:{i}", f"{ak}k",
        ]
        var_map.append(f"v:{i},a:{i}")
    args += [
        "-f", "hls", "-hls_time", str(SEG_SECONDS), "-hls_playlist_type", "vod",
        "-hls_segment_type", "fmp4", "-master_pl_name", "master.m3u8",
        "-var_stream_map", " ".join(var_map),
        str(out_dir / "v%v" / "stream.m3u8"),
    ]
    subprocess.run(args, check=True)
    return out_dir / "master.m3u8"

Failure mode it prevents: buffer underruns and failed adaptive switches. When variants are cut on misaligned boundaries — the usual result of letting the encoder pick scene-cut keyframes independently per rendition — a player cannot switch cleanly and either stalls or plays a glitch at the splice. Forcing a fixed GOP equal to the segment duration (-g, -keyint_min, -sc_threshold 0) makes every segment independently decodable and every variant switchable at every edge. Getting the segment duration right is a tuning problem in its own right, traded off against seek latency and manifest size in the packaging cluster.

Chapter and Metadata Embedding

The second gate makes chapters and metadata durable. A chapter list is generated far upstream — the Transcription & Speaker Diarization layer produces timestamped segments, and automated chapter generation turns those into titled markers — but a marker that exists only in a database is invisible to a podcast app. Chapters have to be written into the deliverable, in the specific atoms and tags each consumer reads, or referenced from a sidecar JSON document the feed points at. Implementing Chapter and Metadata Embedding means writing Nero-style chpl and QuickTime chap chapter atoms into MP4 deliverables with mp4chaps, stamping ID3v2 chapter frames into MP3 renditions, embedding cover artwork at the correct dimensions, and producing the Podcasting 2.0 JSON chapters document referenced by podcast:chapters in the feed. The metadata contract is unforgiving: an artwork frame at the wrong resolution, or a chapter timestamp past the media duration, is enough to make an app silently drop the whole chapter track.

# python 3.10+
# Bento4 1.6.0 (mp4chaps); source chapters are (start_seconds, title) tuples
import subprocess
from pathlib import Path

def write_chapter_file(chapters: list[tuple[float, str]], path: Path) -> None:
    """Emit an mp4chaps sidecar: 'HH:MM:SS.mmm Title' per line, monotonic order."""
    lines = []
    for start, title in sorted(chapters):
        h, rem = divmod(int(start), 3600)
        m, s = divmod(rem, 60)
        ms = int((start - int(start)) * 1000)
        lines.append(f"{h:02d}:{m:02d}:{s:02d}.{ms:03d} {title}")
    path.write_text("\n".join(lines) + "\n", encoding="utf-8")

def embed_chapters(deliverable: Path, chapters: list[tuple[float, str]]) -> None:
    """Write the sidecar next to the file, then import it into the MP4 atoms."""
    chapter_file = deliverable.with_suffix(".chapters.txt")
    write_chapter_file(chapters, chapter_file)
    # mp4chaps reads <name>.chapters.txt and writes chpl + chap atoms in place.
    subprocess.run(["mp4chaps", "--import", str(deliverable)], check=True)

Failure mode it prevents: broken chapter markers and lost artwork. The two most common defects are non-monotonic or out-of-range chapter timestamps — which cause apps to render markers in the wrong place or discard them entirely — and artwork embedded at a resolution outside the 1400×1400 to 3000×3000 pixel window Apple enforces, which strips the image from the episode listing. Sorting timestamps, clamping the last chapter to the media duration, and validating artwork dimensions before the mp4chaps import turns a class of silent app-side failures into a pre-publish contract check.

Podcast RSS Feed Automation

The third gate builds the document the world subscribes to and refuses to publish it if it is malformed. A podcast feed is an RSS 2.0 XML document extended with the iTunes and Podcasting 2.0 namespaces, and it is validated by machines you do not control: Apple’s directory validator, Spotify’s ingestion, and dozens of third-party apps, each stricter than the RSS spec itself. Implementing Podcast RSS Feed Automation means generating that XML deterministically from the episode database, populating every required channel and item element, and — the step teams skip and regret — validating the result before it replaces the live feed. The single highest-value check is confirming that each <enclosure length="..."> matches the actual byte size of the object the CDN will serve, because that one integer, wrong, degrades seeking and download progress in a way no feed linter flags.

# python 3.10+
# feedgen 1.0.0; requests 2.31.0
from feedgen.feed import FeedGenerator
import requests

def build_feed(channel: dict, episodes: list[dict]) -> bytes:
    fg = FeedGenerator()
    fg.load_extension("podcast")
    fg.title(channel["title"])
    fg.link(href=channel["site"], rel="alternate")
    fg.description(channel["description"])
    fg.podcast.itunes_image(channel["artwork_url"])  # 1400-3000 px square
    for ep in episodes:
        fe = fg.add_entry()
        fe.id(ep["guid"])
        fe.title(ep["title"])
        # length MUST equal the served object size; verify against the CDN.
        head = requests.head(ep["enclosure_url"], timeout=10)
        head.raise_for_status()
        served_len = int(head.headers["Content-Length"])
        if served_len != ep["byte_length"]:
            raise ValueError(f"enclosure length mismatch for {ep['guid']}: "
                             f"declared {ep['byte_length']} != served {served_len}")
        fe.enclosure(ep["enclosure_url"], str(served_len), "audio/mpeg")
        fe.podcast.itunes_duration(ep["duration_seconds"])
    return fg.rss_str(pretty=True)

Failure mode it prevents: platform feed rejection and enclosure length mismatch. A feed that omits a required element, carries a relative enclosure URL, or declares a byte length that disagrees with the served object is either rejected outright by Apple and Spotify or accepted and then subtly broken in clients. Verifying the served Content-Length at build time — after the object is on the CDN, not before — closes the gap between what the feed promises and what the edge delivers. Chapter-aware feeds add a further requirement covered in the feed cluster: the podcast:chapters element must reference a reachable JSON document with the correct MIME type.

CDN Distribution and Cache Invalidation

The fourth gate puts the bytes where listeners fetch them and makes sure the edge forgets the old version. A CDN is what lets one origin serve millions of downloads, but its entire value proposition — caching — is also the mechanism by which a corrected episode fails to reach anyone. Implementing CDN Distribution and Cache Invalidation means uploading segments, deliverables, artwork, and the feed to origin with correct cache-control headers, then, on any republish, invalidating precisely the surrogate keys that changed — the feed and the mutated objects — without purging the entire distribution and triggering an origin-fetch stampede. The discipline is to make immutable objects (content-addressed segments) cacheable forever and mutable objects (the feed) short-lived or explicitly purged, so that invalidation is surgical rather than a blunt full-cache flush.

# python 3.10+
# requests 2.31.0; Fastly-style surrogate-key purge API
import requests

CDN_API = "https://api.cdn.example/service/{sid}/purge"

def publish_and_purge(sid: str, token: str, objects: list[dict],
                      changed_keys: set[str]) -> None:
    """Upload with cache headers, then purge only the surrogate keys that changed."""
    for obj in objects:
        # Segments are immutable -> cache forever; the feed is volatile -> revalidate.
        cache_control = ("public, max-age=31536000, immutable"
                         if obj["immutable"] else "public, max-age=60")
        requests.put(
            obj["origin_url"], data=obj["body"], timeout=30,
            headers={"Cache-Control": cache_control,
                     "Surrogate-Key": obj["surrogate_key"]},
        ).raise_for_status()
    # Purge by key, not by URL: one call clears every edge copy of the changed set.
    for key in sorted(changed_keys):
        requests.post(
            CDN_API.format(sid=sid),
            headers={"Fastly-Key": token, "Surrogate-Key": key},
            timeout=15,
        ).raise_for_status()

Failure mode it prevents: stale episodes served from the edge and byte-range 416 errors. When a republish mutates an object in place without invalidation, edges keep serving the old bytes until the TTL lapses — and if the new object has a different length than the cached range clients already recorded, range requests return 416 Range Not Satisfiable and downloads break mid-stream. Content-addressing immutable objects so their URLs never collide, and purging the feed and any genuinely mutated keys by surrogate key, guarantees the next fetch after a republish sees the corrected episode. Purge APIs are rate-limited, so a large republish batches its invalidations rather than firing one call per object.

Compute Handoff & Scaling

Delivery has a different compute profile from the layers before it. Ingestion and transcription are dominated by decode and inference; delivery is dominated by two things — CPU-bound packaging fan-out and I/O-bound upload — and its scaling bottlenecks are almost always external rate limits rather than local horsepower. Packaging a video ladder is embarrassingly parallel across renditions: each rung of the ladder is an independent ffmpeg encode, so a single episode fans out to as many CPU workers as there are ladder steps, and a batch of episodes fans out further. This is the same named-queue discipline the Pipeline Automation & Batch Processing layer applies elsewhere — bind packaging workers to a cpu-package routing key, distinct from the gpu-transcode pool ingestion uses, so a backlog of 1080p encodes never starves the cheap audio-only remuxes waiting behind them.

Upload and purge, by contrast, are throttled by the CDN, not by the worker. Segment upload benefits from high concurrency — a video episode can be hundreds of small fMP4 objects — but a naive fan-out of one HTTP connection per object saturates the origin’s connection pool and trips per-account request ceilings. The workable pattern is a bounded connection pool (a semaphore around a shared requests.Session or an asyncio gather with a concurrency cap) sized to the CDN’s documented limit, with multipart upload for objects above a few megabytes. Cache purge is the tightest constraint of all: purge-by-key APIs enforce a low request-per-second ceiling, so a republish that touches many objects must group them under a shared surrogate key and issue one purge, not one purge per object. When a batch republish would exceed the purge rate limit even after grouping, the correct move is to back off and coalesce — never to fall back to a full-distribution purge, which is unmetered but triggers an origin-fetch stampede across every edge at once.

Observability Checklist

Delivery failures are expensive precisely because they are public and often silent — a stale edge or a one-byte enclosure mismatch produces no exception, only a degraded listener experience. Instrument every gate with the following Prometheus metrics, emit the listed structured log fields on every publish, and wire the alerting thresholds before the layer carries production volume.

• delivery_package_seconds{rendition} (histogram) — wall time to encode one ladder rung. Alert when the 1080p p95 exceeds 4× real-time; a regression here usually means a lost hardware-encode path or an ffmpeg filter added without a preset change.
• feed_validation_failures_total{check} (counter) — feed builds rejected by a pre-publish check, labeled by the failing check (schema, enclosure_length, artwork_dims, relative_url). Any nonzero rate on schema or relative_url should page — those feeds would have been rejected by Apple or Spotify.
• enclosure_length_mismatch_total (counter) — enclosures whose declared length disagreed with the served Content-Length. This must be zero in steady state; a single increment means the feed and the CDN object diverged and seeking is broken for that episode.
• cdn_purge_latency_seconds (histogram) — time from purge request to confirmed edge eviction. Alert when p95 exceeds 30 s; slow purges extend the window during which listeners get a stale episode after a republish.
• cdn_purge_ratelimit_total (counter) — purge calls rejected with a rate-limit response. A sustained rise means republish batches are not coalescing surrogate keys and are about to be throttled into a stale-content incident.

Every structured log line should carry episode_id, feed_id, gate (package/embed/feed/distribute), result, error_code, enclosure_bytes, surrogate_key, and duration_ms. With those fields you can reconstruct any episode’s full path through the delivery pipeline, join a listener-reported stale episode to the exact purge that should have evicted it, and trace an enclosure mismatch back to the packaging step whose output length changed. Carry the same episode_id the upstream layers stamped so a delivery incident joins cleanly to the ingestion and transcription decisions that produced the asset.

Failure Modes Reference

The table below consolidates the failure classes this layer is built to contain, mapping each to its public-facing symptom, root cause, and the deterministic remediation the delivery pipeline applies. It is the on-call reference for triaging a publish that shipped wrong or a listener report of a broken episode.

Related

• Adaptive Bitrate Packaging — keyframe-aligned HLS/DASH rendition ladders from a normalized master.
• Chapter and Metadata Embedding — durable chapter atoms, ID3 frames, and artwork in the deliverable.
• Podcast RSS Feed Automation — deterministic feed generation with pre-publish schema and enclosure validation.
• CDN Distribution and Cache Invalidation — cache-control strategy and surgical surrogate-key purges.
• Media Ingestion & Format Architecture — the ingestion control plane that produces the normalized master this layer packages.
• Transcription & Speaker Diarization — the inference layer whose transcripts and chapters this layer embeds and publishes.