Automated Chapter Generation

Within the Transcription & Speaker Diarization pipeline, automated chapter generation is the stage that turns a finished transcript — words and segments with millisecond timestamps, plus speaker labels — into an ordered list of chapter markers, each with a start time and a human-readable title, without an editor touching the file. It consumes the transcript emitted upstream and produces a small, strict artifact: a chapters JSON array whose start times are embedded later by the delivery layer into the MP4 container and the podcast feed. The stage never re-transcribes and never re-diarizes; it treats the transcript as an immutable input contract and decides only where topics change and what to call them.

Chaptering is deceptively easy to do badly. Naive approaches split on fixed time intervals, on every speaker change, or on the first keyword match — and produce markers that land mid-sentence, mid-word, or three seconds before a laugh with no relationship to the actual structure of the conversation. Doing it well means three separable decisions: segmenting the transcript into topics, snapping each topic boundary to a natural acoustic and grammatical break, and naming the resulting spans. This component sits after transcription and diarization complete and before packaging, so its output must be deterministic, auditable, and cheap to regenerate when a transcript is corrected.

Prerequisites & Environment

Pin every dependency. Embedding models and their tokenizers change segmentation boundaries between minor versions, so an unpinned model is a reproducibility hazard — the same episode can chapter differently on two workers.

• Python: 3.10+ (the examples use structural pattern matching and list/dict generics).
• Python libraries (pinned):

# sentence-transformers==3.0.1   # sentence embeddings for topic segmentation
# numpy==1.26.4                  # cosine-similarity / drift math
# pydantic==2.7.1                # validates the transcript and chapter contracts
# scikit-learn==1.5.0            # optional: peak detection on the drift curve

• Model: sentence-transformers/all-MiniLM-L6-v2 is the default segmenter embedding — 384-dimensional, ~90 MB, CPU-friendly at roughly 2,000 sentences/second on a single core. A larger all-mpnet-base-v2 gives marginally cleaner boundaries at ~4x the cost; keep the choice in configuration.
• Hardware: segmentation is CPU-bound and embarrassingly parallel across episodes. Budget one core and ~400 MB RAM per concurrent job (model weights plus the embedding matrix for a long-form episode). No GPU is required unless you swap in a large transformer titler.
• Environment variables the worker reads at startup:

CHAP_EMBED_MODEL=sentence-transformers/all-MiniLM-L6-v2
CHAP_DRIFT_THRESHOLD=0.35     # cosine-distance peak height that marks a topic change
CHAP_SNAP_WINDOW_MS=2500      # search radius when snapping a boundary to a real break
CHAP_MIN_LEN_S=60             # minimum chapter duration in seconds
CHAP_MAX_CHAPTERS=30          # hard ceiling on markers per episode
CHAP_TITLER=llm              # llm | extractive

The two load-bearing knobs are CHAP_DRIFT_THRESHOLD (how eagerly the segmenter declares a topic change) and CHAP_MIN_LEN_S (how aggressively short chapters are merged away). Bake both into configuration per show format — a fast-paced news roundup wants a lower minimum than a two-hour interview.

Architecture & Queue Topology

The chapter worker pulls a completed transcript from an upstream queue, runs a strict segment → snap → title pipeline, and writes a chapters JSON sidecar back to object storage. Speaker turns come from speaker diarization with Pyannote, and any word timestamps that drifted during transcription are already reconciled by timestamp alignment and correction before this stage runs — the segmenter trusts the timestamps it is handed and never re-measures them.

The worker is stateless per episode: it holds no cross-episode memory, and the chapters JSON is a pure function of the transcript plus the configuration tuple. That makes every job idempotent — re-running a corrected transcript overwrites the sidecar deterministically.

Step-by-Step Implementation

Each step ends with a verification command you can run before promoting the artifact.

1. Load and validate the transcript + diarization contract

Before any segmentation, validate the input. A missing timestamp or an out-of-order segment here would produce silently wrong boundaries downstream, so reject malformed transcripts at the cheapest possible point.

# pydantic==2.7.1
from pydantic import BaseModel, field_validator

class Word(BaseModel):
    text: str
    start_ms: int
    end_ms: int
    speaker: str | None = None

class TranscriptContract(BaseModel):
    words: list[Word]
    duration_ms: int

    @field_validator("words")
    @classmethod
    def monotonic(cls, ws: list[Word]) -> list[Word]:
        for a, b in zip(ws, ws[1:]):
            if b.start_ms < a.start_ms:
                raise ValueError("word timestamps must be non-decreasing")
        return ws

Verify the transcript parses and its timestamps are ordered:

python -c "import chapters,json; t=chapters.TranscriptContract(**json.load(open('transcript.json'))); print(len(t.words), 'words ok')"

2. Window and embed the transcript

Topic segmentation works on units larger than a word. Group words into sentences (or fixed ~25-word windows if punctuation is unreliable), then embed each window. The embedding matrix is the raw material for every boundary decision.

# sentence-transformers==3.0.1  numpy==1.26.4
import os
import numpy as np
from sentence_transformers import SentenceTransformer

_model = SentenceTransformer(os.environ["CHAP_EMBED_MODEL"])

def embed_windows(windows: list[str]) -> np.ndarray:
    # normalize_embeddings=True lets us use a plain dot product as cosine similarity
    return _model.encode(windows, normalize_embeddings=True, batch_size=64)

Verify the embedding matrix has one row per window and unit-norm rows:

python -c "import chapters,numpy as np; e=chapters.embed_windows(['a topic','another']); print(e.shape, np.allclose(np.linalg.norm(e,axis=1),1))"

3. Score semantic drift and pick candidate boundaries

A topic change shows up as a dip in similarity between adjacent windows — equivalently, a peak in cosine distance. This is the embedding-based analogue of the classic TextTiling algorithm: smooth the adjacent-window similarity into a depth score and mark local minima that fall below the drift threshold. Each qualifying gap becomes a candidate boundary.

# numpy==1.26.4
import numpy as np

def candidate_boundaries(emb: np.ndarray, threshold: float) -> list[int]:
    # similarity between window i and i+1
    sim = np.sum(emb[:-1] * emb[1:], axis=1)
    dist = 1.0 - sim
    # smooth to suppress single-window noise
    k = np.array([0.25, 0.5, 0.25])
    smooth = np.convolve(dist, k, mode="same")
    cuts = []
    for i in range(1, len(smooth) - 1):
        is_peak = smooth[i] >= smooth[i - 1] and smooth[i] > smooth[i + 1]
        if is_peak and smooth[i] >= threshold:
            cuts.append(i + 1)   # boundary sits *before* window i+1
    return cuts

Verify the boundary count is sane for the episode — a two-hour interview should not produce 200 candidates:

python -c "import chapters; print('candidates:', chapters.count_candidates('transcript.json'))"

4. Snap boundaries to silence, sentence end, and speaker turn

A candidate boundary from step 3 points at a window index, not a clean time. Snapping moves it to the nearest natural break inside CHAP_SNAP_WINDOW_MS: prefer a silence gap, then a sentence-final punctuation mark, then a speaker turn from diarization. A boundary that lands mid-sentence is the single most common chaptering defect, and this step exists to prevent it.

# python 3.10+
def snap_boundary(words: list, idx: int, window_ms: int) -> int:
    """Return a start_ms snapped to the best break near words[idx]."""
    target = words[idx].start_ms
    best_ms, best_rank = target, 99
    for w_prev, w in zip(words, words[1:]):
        gap = w.start_ms - w_prev.end_ms
        if abs(w.start_ms - target) > window_ms:
            continue
        rank = (0 if gap >= 400                       # a real silence
                else 1 if w_prev.text.endswith((".", "?", "!"))
                else 2 if w.speaker != w_prev.speaker  # speaker turn
                else 9)
        if rank < best_rank:
            best_rank, best_ms = rank, w.start_ms
    return best_ms

The precise mapping from a snapped boundary to a chapter start time — including the millisecond conversion and the JSON emission that both downstream consumers read — is documented in mapping transcript timestamps to chapter markers. Verify each snapped start coincides with a word boundary:

python -c "import chapters; chapters.assert_starts_on_words('transcript.json'); print('all starts land on words')"

5. Enforce minimum chapter length and cap the count

Segmentation and snapping can leave chapters that are too short to be useful — a 12-second interstitial is noise in a player’s chapter list. Merge any span shorter than CHAP_MIN_LEN_S into its neighbour, then, if the episode still exceeds CHAP_MAX_CHAPTERS, drop the shallowest boundaries (lowest drift score) until it fits.

# python 3.10+
def enforce_min_length(starts_ms: list[int], duration_ms: int, min_s: int) -> list[int]:
    min_ms = min_s * 1000
    kept = [starts_ms[0]]
    for s in starts_ms[1:]:
        if s - kept[-1] >= min_ms:
            kept.append(s)
    # never leave a final chapter shorter than the minimum
    if len(kept) > 1 and duration_ms - kept[-1] < min_ms:
        kept.pop()
    return kept

Verify no adjacent starts are closer than the minimum:

python -c "import chapters; chapters.assert_min_gap('chapters.json', 60); print('min-length ok')"

6. Title each chapter and emit the chapters JSON

The final step names each span from its transcript text and writes the sidecar. The titler is pluggable: an extractive titler picks the most representative sentence, while the LLM titler generates a concise label — the LLM-assisted chapter titling from transcripts approach is the higher-quality path, with the extractive titler as its deterministic fallback.

# pydantic==2.7.1
import json, pathlib
from pydantic import BaseModel

class Chapter(BaseModel):
    start_ms: int
    end_ms: int
    title: str
    confidence: float

def emit_chapters(chapters: list[Chapter], dst: str) -> str:
    payload = [c.model_dump() for c in chapters]
    pathlib.Path(dst).write_text(json.dumps(payload, indent=2))
    return dst

Verify the emitted file is valid JSON with monotonic, non-overlapping spans:

jq -e 'all(.[]; .start_ms < .end_ms) and (. == sort_by(.start_ms))' chapters.json

Data Contracts

The output is an ordered array of chapter objects. Downstream stages trust this contract instead of inspecting the transcript, so every field is validated before the sidecar is written.

The first chapter always starts at 0. end_ms is derived, never authored independently — it is the next chapter’s start or the episode duration — which guarantees the array tiles the timeline with no gaps and no overlaps. The confidence field lets the delivery layer or a human reviewer sort weak markers for spot-checking without re-running the pipeline.

Resilience Patterns

Chapter jobs are pure functions of their transcript and configuration, which keeps resilience straightforward.

• Idempotency. Key the job on the SHA-256 of the transcript plus the configuration tuple. A retried job that finds a matching sidecar short-circuits and returns it — no recomputation, no divergent boundaries.
• Titler isolation. The LLM titler is the only stage that reaches outside the process. Wrap it in a timeout and a circuit breaker: on repeated failures, fall back to the extractive titler so the pipeline still ships chapters with weaker titles rather than blocking the episode.
• Degradation over failure. If segmentation produces zero boundaries (a monologue with no topic drift), emit a single full-length chapter titled from the episode metadata rather than failing. An empty chapters array breaks downstream embedding.
• Partial-write safety. Write the sidecar to a temp path and atomically rename on success, so a crashed titler never leaves a half-written JSON that a downstream reader might pick up.
• Correction reprocessing. When a transcript is corrected upstream, the changed SHA invalidates the sidecar and the episode re-chapters automatically. Because timestamps were already reconciled by the alignment stage, boundaries stay stable unless the correction actually changed the words.

Observability & Debugging

Instrument the worker so bad chaptering is visible before a listener sees it.

• Key metrics (Prometheus):
  • chap_chapters_total — histogram of chapters per episode, labeled by show; a sudden spike usually means CHAP_DRIFT_THRESHOLD is too low.
  • chap_short_merged_total — counter of chapters merged for falling under CHAP_MIN_LEN_S.
  • chap_titler_fallback_total — counter incremented each time the LLM titler timed out and the extractive fallback ran; alert if the rate climbs.
  • chap_segment_seconds — histogram of end-to-end segmentation time per episode.
• Structured log fields: episode_id, transcript_sha, n_candidates, n_final, n_merged, titler, drift_threshold, duration_ms.
• Common errors and root causes:
  • Boundaries land mid-sentence — the snap window is too small; raise CHAP_SNAP_WINDOW_MS or confirm diarization speaker turns are present.
  • Too many tiny chapters — drift threshold too low or minimum length too short; raise CHAP_DRIFT_THRESHOLD toward 0.4.
  • One giant chapter on a varied episode — embeddings collapsed (wrong model loaded, or all-caps transcript); confirm the pinned model and normalize casing before embedding.

Performance Tuning

• Batch the embeddings. The dominant cost is model.encode; raise batch_size to 128 on machines with spare memory and encode the whole episode in one call rather than per-window.
• Cache the model. Load the SentenceTransformer once at worker startup, not per job — cold-loading 90 MB of weights per episode dominates latency at scale.
• Parallelize across episodes, not within. Segmentation is single-threaded and cheap per episode; a ProcessPoolExecutor sized to the core count processes a back catalogue far faster than trying to thread one episode.
• Co-schedule with CPU-idle stages. Because chaptering is CPU-bound and GPU-free, it co-schedules cleanly on the same hosts as GPU-bound transcription work, filling otherwise-idle cycles.
• Skip the titler on regeneration. If only boundaries changed and a chapter’s span is unchanged, reuse the cached title instead of re-calling the LLM — title generation is the most expensive stage per chapter.

Frequently Asked Questions

Related

• Transcription & Speaker Diarization — the parent pipeline this stage belongs to.
• Speaker diarization with Pyannote — supplies the speaker turns used as snap targets.
• Timestamp alignment and correction — reconciles word timestamps before this stage trusts them.
• Mapping transcript timestamps to chapter markers — the precise boundary-to-start-time conversion.
• LLM-assisted chapter titling from transcripts — the higher-quality titler for the final step.
• Chapter and metadata embedding — the delivery consumer that embeds the chapters JSON into the container.
• Podcast RSS feed automation — the delivery consumer that writes chapter tags into the feed.