Aligning Speaker Labels with Video Cuts

Within the transcription and speaker diarization pipeline, and downstream of the timestamp alignment and correction stage, this technique solves one precise production scenario: multi-camera interviews and podcast video where audio-derived speaker turns refuse to land on the frame-accurate boundaries an editor actually cut on, so lower-thirds, B-roll overlays, and auto-chapters render on the wrong subject. The fix is a deterministic reconciliation pass that snaps probabilistic diarization output onto rigid edit-decision-list boundaries.

Problem Framing

Automated transcription emits speaker turns at acoustic boundaries; a video editor cuts at frame boundaries. The two grids almost never coincide. A diarization model places a speaker change at 00:04:12.137; the NLE cut sits at frame 6051 of a 24fps timeline (00:04:12.125). Twelve milliseconds of disagreement is inaudible, but it is enough to render a lower-third graphic naming the wrong person for one frame on either side of the cut, or to start an auto-chapter mid-sentence.

In a running pipeline the symptom surfaces as a steady drizzle of off-by-a-frame labels rather than a hard crash. A worker processing a two-camera interview emits a manifest where a speaker turn straddles a scene transition:

WARN  aligner | segment SPK_02 [252137..258960] crosses cut SCENE_07@252125 by +12ms
WARN  aligner | lower-third for SPK_02 rendered over SPK_01 frame at 00:04:12.125
INFO  render  | chapter "Q3 roadmap" anchored mid-utterance at 00:04:12.150

Left unhandled, three failure shapes recur: labels bleed across cuts so graphics name the wrong subject; micro-fragments (a 180ms “yeah” between two long turns) generate flickering overlays the rendering engine chokes on; and overlapping turns from the same speaker, split by the diarizer, produce duplicate adjacent captions. None of these throw — they ship, and an editor finds them in QC. The reconciliation pass exists to make the audio grid conform to the video grid before the manifest leaves the pipeline.

Solution Architecture

The aligner is a stateless post-processing service that ingests three artifacts: frame-accurate cut timestamps (extracted from an EDL/XML export or ffprobe scene-change detection), the diarization segments produced by speaker diarization with Pyannote, and the word-level timing anchors from Whisper Large V3 integration. It runs strictly after base audio drift has been normalized against the master clock, and it is dispatched through the same async transcription queue so it never blocks the ingestion worker.

The logic is three deterministic phases applied per segment:

  1. Boundary snapping — if a segment start or end falls within the tolerance window of a cut, the timestamp is pinned to the exact cut frame, so a speaker label can never bleed across a scene transition.
  2. Overlap resolution — when adjacent turns belong to the same speaker and the gap between them is below the merge threshold, they collapse into one turn; higher-confidence outputs win contested windows.
  3. Sub-threshold filtering — fragments shorter than the minimum viable duration are dropped before they reach the renderer.

Boundary lookups use binary search over the sorted cut list for O(log n) cost, so the pass scales to feature-length timelines with thousands of cuts.

Reconciling the audio grid with the video grid Two input grids feed a stateless aligner: diarization segments whose edges fall between video frames, and frame-accurate cut points. The aligner snaps edges to boundaries, merges same-speaker turns, and drops sub-threshold fragments, emitting one manifest whose labels sit exactly on cut frames. Diarization segments acoustic boundaries · SPK ids + confidence SPK_01 SPK_02 edges fall between frames (+12 ms) Video cut frames frame grid · EDL / ffprobe scene cuts SCENE_06 SCENE_07 SCENE_08 252125 ms · frame 6051 @ 24fps Stateless cut aligner bisect · O(log n) · thread-safe 1 · Snap to boundary pin edge to cut within tolerance window 2 · Merge same-speaker collapse gaps below merge threshold 3 · Drop sub-threshold discard fragments below min duration Reconciled manifest labels sit exactly on cut frames SPK_01 SPK_02 is_snapped ✓ · anchored to SCENE_07 ledger: snapped · merged · dropped

Implementation

The aligner below is stateless and thread-safe: every call is a pure transformation of (cuts, segments) into an aligned manifest, so it is safe to fan out across workers and safe to retry under at-least-once delivery. It carries a diagnostic ledger (snapped / merged / dropped) for observability and uses bisect for the O(log n) boundary lookups. It depends only on the Python 3.10+ standard library — no pinned third-party versions required.

# Python 3.10+; standard library only (bisect, logging, dataclasses, typing)
import bisect
import logging
from dataclasses import dataclass
from typing import List, Optional

logging.basicConfig(level=logging.INFO, format="%(asctime)s | %(levelname)-8s | %(message)s")
logger = logging.getLogger(__name__)

@dataclass
class CutPoint:
    timestamp_ms: int   # frame-accurate cut boundary in milliseconds
    scene_id: str       # stable id from the EDL / scene-detection export

@dataclass
class DiarizationSegment:
    start_ms: int
    end_ms: int
    speaker_id: str
    confidence: float   # 0.0-1.0, from the diarizer

@dataclass
class AlignedSegment:
    start_ms: int
    end_ms: int
    speaker_id: str
    cut_boundary: Optional[str]  # scene_id this turn is anchored to
    is_snapped: bool             # True if either edge was pinned to a cut

class SpeakerCutAligner:
    def __init__(
        self,
        cut_tolerance_ms: int = 150,        # max distance an edge may be pulled to a cut
        min_segment_duration_ms: int = 800, # turns shorter than this are dropped
        max_overlap_merge_ms: int = 300     # same-speaker gaps below this are merged
    ):
        self.cut_tolerance_ms = cut_tolerance_ms
        self.min_segment_duration_ms = min_segment_duration_ms
        self.max_overlap_merge_ms = max_overlap_merge_ms
        self._diagnostics = {"total_segments": 0, "snapped": 0, "merged": 0, "dropped": 0}

    def align(self, cuts: List[CutPoint], segments: List[DiarizationSegment]) -> List[AlignedSegment]:
        if not cuts or not segments:
            raise ValueError("Both cut points and diarization segments are required for alignment.")

        self._diagnostics = {"total_segments": len(segments), "snapped": 0, "merged": 0, "dropped": 0}
        cut_timestamps = sorted(c.timestamp_ms for c in cuts)   # sorted once for binary search
        cut_map = {c.timestamp_ms: c.scene_id for c in cuts}

        aligned: List[AlignedSegment] = []
        pending_merge: Optional[AlignedSegment] = None

        # Process in time order so merge decisions only look at the previous turn.
        for seg in sorted(segments, key=lambda s: s.start_ms):
            try:
                snapped_seg = self._snap_to_boundary(seg, cut_timestamps, cut_map)

                # Phase 3: drop micro-fragments before they reach the renderer.
                if (snapped_seg.end_ms - snapped_seg.start_ms) < self.min_segment_duration_ms:
                    self._diagnostics["dropped"] += 1
                    logger.debug(f"Dropping sub-threshold segment: {snapped_seg}")
                    continue

                # Phase 2: collapse adjacent same-speaker turns.
                if pending_merge and self._can_merge(pending_merge, snapped_seg):
                    pending_merge = self._merge_segments(pending_merge, snapped_seg)
                    self._diagnostics["merged"] += 1
                else:
                    if pending_merge:
                        aligned.append(pending_merge)
                    pending_merge = snapped_seg

            except Exception as e:
                logger.error(f"Alignment failed for segment {seg}: {e}")
                continue

        if pending_merge:
            aligned.append(pending_merge)

        logger.info(f"Alignment complete. Diagnostics: {self._diagnostics}")
        return aligned

    def _snap_to_boundary(self, seg: DiarizationSegment, cut_ts: List[int], cut_map: dict) -> AlignedSegment:
        start, end = seg.start_ms, seg.end_ms
        snapped_start, snapped_end = start, end
        is_snapped = False

        # Phase 1a: pull the start edge to the nearest cut within tolerance.
        idx = bisect.bisect_left(cut_ts, start)
        if idx < len(cut_ts) and abs(cut_ts[idx] - start) <= self.cut_tolerance_ms:
            snapped_start, is_snapped = cut_ts[idx], True
        elif idx > 0 and abs(cut_ts[idx - 1] - start) <= self.cut_tolerance_ms:
            snapped_start, is_snapped = cut_ts[idx - 1], True

        # Phase 1b: pull the end edge to the nearest cut within tolerance.
        idx = bisect.bisect_left(cut_ts, end)
        if idx < len(cut_ts) and abs(cut_ts[idx] - end) <= self.cut_tolerance_ms:
            snapped_end, is_snapped = cut_ts[idx], True
        elif idx > 0 and abs(cut_ts[idx - 1] - end) <= self.cut_tolerance_ms:
            snapped_end, is_snapped = cut_ts[idx - 1], True

        # Guard against an inverted interval after snapping both edges to the same cut.
        if snapped_end <= snapped_start:
            snapped_end = snapped_start + self.min_segment_duration_ms

        # Anchor the turn to the scene it sits inside, by its midpoint.
        mid_point = (snapped_start + snapped_end) // 2
        idx = bisect.bisect_left(cut_ts, mid_point)
        boundary_ts = cut_ts[idx - 1] if idx > 0 else cut_ts[0]
        boundary = cut_map.get(boundary_ts)

        self._diagnostics["snapped"] += 1 if is_snapped else 0
        return AlignedSegment(
            start_ms=snapped_start,
            end_ms=snapped_end,
            speaker_id=seg.speaker_id,
            cut_boundary=boundary,
            is_snapped=is_snapped,
        )

    def _can_merge(self, a: AlignedSegment, b: AlignedSegment) -> bool:
        return (a.speaker_id == b.speaker_id) and ((b.start_ms - a.end_ms) <= self.max_overlap_merge_ms)

    def _merge_segments(self, a: AlignedSegment, b: AlignedSegment) -> AlignedSegment:
        return AlignedSegment(
            start_ms=a.start_ms,
            end_ms=max(a.end_ms, b.end_ms),
            speaker_id=a.speaker_id,
            cut_boundary=a.cut_boundary or b.cut_boundary,
            is_snapped=a.is_snapped or b.is_snapped,
        )

The three constructor parameters are the only tuning surface. cut_tolerance_ms (default 150) is the widest gap the snapper will close — set it just above one frame interval (≈42ms at 24fps, ≈33ms at 30fps) plus expected acoustic jitter. min_segment_duration_ms (default 800) is the floor below which a turn is treated as a fragment. max_overlap_merge_ms (default 300) is how large a same-speaker gap may be and still collapse into one turn. Tighten tolerance for fast-cut edits where adjacent cuts sit closer than the window; loosen it for slow interviews with sparse cuts.

Verification

Confirm three things before promoting the worker: that cut points are genuinely frame-accurate, that the diagnostics ledger is sane, and that the output matches a hand-aligned golden set.

Extract the cut grid and confirm boundaries fall on frame intervals with the official FFmpeg documentation scene filter:

# ffmpeg 6.x: dump frame-change timestamps for the cut grid
ffprobe -v error -show_frames -of csv=p=0 \
  -f lavfi "movie=interview.mp4,select=gt(scene\,0.4)" \
  | awk -F',' '{print $1}'

Run the aligner against a fixture and assert the ledger and the snapping invariant:

# pytest fixture: every emitted edge must sit on a cut or untouched, never between
aligned = SpeakerCutAligner(cut_tolerance_ms=150).align(cuts, segments)
cut_set = {c.timestamp_ms for c in cuts}
for s in aligned:
    assert s.end_ms > s.start_ms                       # no inverted intervals
    if s.is_snapped:
        assert s.start_ms in cut_set or s.end_ms in cut_set
assert all(s.end_ms - s.start_ms >= 800 for s in aligned)  # no fragments survived

A healthy run logs a low dropped count and a snapped count close to the number of true speaker changes. The bisect-based lookups guaranteeing that O(log n) cost are documented in the official Python bisect module; diff every output manifest against the golden set and gate promotion on zero regressions.

Failure Modes & Edge Cases

Edge case Symptom Mitigation
Two cuts inside one tolerance window both edges snap to the same frame, interval collapses inverted-interval guard restores min_segment_duration_ms; lower cut_tolerance_ms below the inter-cut gap
Phantom segment from low-bitrate audio sub-800ms turn between two long turns flickers a lower-third phase-3 filtering drops it; raise min_segment_duration_ms for noisy sources
Diarizer splits one turn at a breath duplicate adjacent captions for the same speaker phase-2 merge collapses gaps under max_overlap_merge_ms
Cut grid in a different frame rate than audio clock systematic snap offset of one frame across the asset normalize both to milliseconds against the master clock before alignment
Segments arrive out of order from the queue merge logic compares non-adjacent turns align() sorts by start_ms first; never assume queue order
Empty cut list (no scene changes detected) ValueError raised at entry route to DLQ and fall back to uncorrected timestamps rather than emit unanchored labels

FAQ

How wide should the cut tolerance window be?

Start at one frame interval plus expected acoustic jitter: ≈42ms at 24fps or ≈33ms at 30fps, padded to roughly 100-150ms for typical diarization drift. If adjacent cuts in a fast-paced edit sit closer than the window, two edges can snap to the same frame — lower cut_tolerance_ms below the minimum inter-cut gap so the inverted-interval guard never has to fire.

Why snap to cut frames instead of trusting the diarizer's timestamps?

The diarizer emits boundaries at acoustic events, which land a few milliseconds off the frame grid the editor actually cut on. Rendering a lower-third or chapter marker against an off-grid timestamp names the wrong subject for a frame on either side of the cut. Pinning each edge to the exact cut frame makes the audio grid conform to the video grid, which is what eliminates the off-by-a-frame labels.

What happens when no cuts are detected at all?

The aligner raises ValueError at entry rather than guessing. In production that exception routes the job to the dead-letter queue with the raw payload preserved, and a downstream consumer falls back to uncorrected timestamps. An unanchored manifest is worse than an honestly-skipped one, so the stage fails closed instead of emitting labels with no scene context.