Handling Overlapping Speech in Diarization

This page sits beneath Speaker Diarization with Pyannote inside the wider Transcription & Speaker Diarization pipeline, and it targets one stubborn failure: two people talking at the same time. A guest interrupts, a co-host laughs over the answer, someone says “yeah, exactly” across the sentence — and the diarization output collapses that region to a single label, dropping one voice entirely or smearing both into whichever speaker had the louder frame. This is the single largest source of attribution error in conversational podcast and panel audio, and no amount of transcript cleanup recovers a speaker the diarizer never emitted.

Problem Framing

Standard diarization treats each frame as belonging to exactly one speaker. pyannote/speaker-diarization-3.1 internally models overlap through its powerset segmentation head, but the region where two speakers are genuinely concurrent still tends to be under-reported once the pipeline resolves clusters into a clean turn timeline. On real crosstalk you see the region attributed to one speaker, and the interjecting speaker’s words arrive in the transcript with no diarization segment to hang them on. The reconciliation log makes the gap obvious:

WARN  attribution  seg=ep131_00:14:22-00:14:25  words=7  matched_speaker=None  reason="no diarization turn covers span"
WARN  overlap      seg=ep131_00:31:08-00:31:11  active_speakers=2  emitted_labels=1  dropped="SPEAKER_02"
INFO  der          file=ep131  DER=18.4%  missed_detection=2.1%  false_alarm=1.3%  confusion=15.0%

That 15% confusion figure is not model failure on clean speech — it is concentrated almost entirely in the seconds where voices overlap. If you compute error separately over overlap and non-overlap regions, the non-overlap DER is often under 5% while the overlap DER exceeds 40%. The fix is not a better clustering threshold; it is to detect the overlapped-speech regions explicitly, attribute the two most-active speakers to them, and carry both labels through to the transcript so that downstream timestamp alignment and correction has two turns to align instead of one truncated guess.

Solution Architecture

The approach runs two Pyannote pipelines over the same waveform. The first is the normal pyannote/speaker-diarization-3.1 pipeline, which produces the per-speaker turn timeline. The second is an OverlappedSpeechDetection pipeline built on pyannote/segmentation-3.0, which produces a binary timeline of “two-or-more speakers active” regions independent of who they are. A reconciler then intersects the two: for every detected overlap region, it looks up which speakers the diarization considered active in a small window around it and promotes the top two into a dual-label segment. The enriched annotation — now carrying overlapping turns — feeds transcript reconciliation and an overlap-aware error measurement.

Dual-pipeline overlap attribution flow in Pyannote A single episode waveform fans out to two pipelines. The upper branch is the speaker-diarization-3.1 pipeline producing a per-speaker turn timeline. The lower branch is an OverlappedSpeechDetection pipeline built on segmentation-3.0 producing binary two-or-more-speaker regions. Both feed an overlap reconciler that, for each overlap region, selects the two most active speakers and writes a dual-label segment. The enriched annotation then splits to two consumers: transcript word reconciliation, which assigns overlapping words to both speakers, and overlap-aware diarization error rate, which scores the run with skip_overlap set to false. Episode waveform 16 kHz mono · crosstalk Diarization pipeline speaker-diarization-3.1 per-speaker turn timeline Overlap detection segmentation-3.0 binary 2+ speaker regions Overlap reconciler top-2 active speakers → dual-label segment enriched annotation enriched annotation Transcript reconciliation both speakers per word Overlap-aware DER skip_overlap=False

Because the two pipelines are independent, you can pin the overlap detector’s sensitivity separately from clustering. Studio recordings with clean turn-taking rarely trip the detector, so the reconciler is a no-op on them; only genuinely overlapped regions ever gain a second label, which keeps the non-overlap DER exactly where the base pipeline left it.

Implementation

1. Instantiate both pipelines

Pin pyannote.audio==3.1.1 and accept the model license on the Hugging Face model pages before first run. The diarization pipeline gives you the turn timeline; the OverlappedSpeechDetection pipeline gives you the where-two-voices-collide timeline.

# pyannote.audio==3.1.1  torch==2.3.1  pyannote.core==5.0.0  pyannote.metrics==3.2.1
import logging
import torch
from pyannote.audio import Pipeline, Model
from pyannote.audio.pipelines import OverlappedSpeechDetection
from pyannote.core import Segment, Annotation

logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
logger = logging.getLogger(__name__)

HF_TOKEN = "hf_your_read_token"   # accept model licenses first on huggingface.co
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Base diarization: emits the per-speaker turn timeline.
diarization = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1", use_auth_token=HF_TOKEN
).to(DEVICE)

# Overlapped-speech detection: binary "2+ speakers active" timeline.
osd_model = Model.from_pretrained("pyannote/segmentation-3.0", use_auth_token=HF_TOKEN)
osd = OverlappedSpeechDetection(segmentation=osd_model)
osd.instantiate({
    "min_duration_on": 0.10,    # ignore overlaps shorter than 100 ms (clicks, plosives)
    "min_duration_off": 0.10,   # bridge gaps under 100 ms so one interruption stays one region
})
osd.to(DEVICE)

The two thresholds are the load-bearing knobs. min_duration_on of 0.10 discards sub-100 ms blips that are almost always breath or plosive bleed, not real crosstalk; raise it toward 0.25 on noisy field audio to suppress false overlaps. min_duration_off of 0.10 prevents a single interruption from fragmenting into a stutter of tiny regions.

2. Attribute both active speakers to each overlap region

Run both pipelines, then for every overlap region select the two speakers the diarization considered most active across that span and write them into the annotation. crop restricts the diarization to the region; the two longest-covering labels become the concurrent pair.

def enrich_with_overlap(audio_path: str) -> Annotation:
    """Return a diarization Annotation with dual-speaker labels on overlap regions."""
    dia: Annotation = diarization(audio_path)
    osd_out = osd(audio_path)                     # Annotation of "overlap" regions

    enriched = dia.copy()
    for overlap_region in osd_out.get_timeline().support():
        # How much each speaker covers this exact overlap span.
        local = dia.crop(overlap_region, mode="intersection")
        coverage = {
            label: sum(seg.duration for seg in local.label_timeline(label))
            for label in local.labels()
        }
        top_two = sorted(coverage, key=coverage.get, reverse=True)[:2]
        if len(top_two) < 2:
            logger.warning(
                f"Overlap {overlap_region} has <2 diarized speakers "
                f"({top_two}); leaving single label."
            )
            continue
        # Ensure BOTH speakers own the full overlap span (base pipeline kept only one).
        for label in top_two:
            enriched[overlap_region, f"overlap_{label}"] = label
        logger.info(f"Overlap {overlap_region}: attributed {top_two}")
    return enriched

The mode="intersection" crop is what makes this defensible: coverage is measured only inside the overlap span, so a speaker who dominates the surrounding conversation but is silent during the collision does not win the second slot by accident.

3. Reconcile dual-speaker regions with the transcript

Words whose midpoint falls inside an overlap region should carry both speaker labels rather than being force-assigned to one. This keeps interruptions readable and stops a “yeah” from being stolen from the speaker who actually said it.

def assign_speakers_to_words(words: list, enriched: Annotation) -> list:
    """Attach one or two speaker labels to each transcript word by timestamp."""
    for w in words:
        mid = Segment((w["start"] + w["end"]) / 2, (w["start"] + w["end"]) / 2 + 1e-3)
        active = {label for seg, _, label in enriched.itertracks(yield_label=True)
                  if seg.overlaps(mid)}
        w["speakers"] = sorted(active) if active else ["UNKNOWN"]
        if len(w["speakers"]) > 1:
            logger.info(f"Word '{w['text']}' at {w['start']:.2f}s -> overlap {w['speakers']}")
    return words

Word timestamps here come from the transcription engine (Whisper or a hosted API). If those word boundaries drift relative to the diarization frames, correct them through the timestamp alignment and correction stage before this reconciliation runs, or the overlap window will land on the wrong words.

Verification

Confirm each stage in isolation, then score the whole run against a reference RTTM with overlap counted in.

# 1. Confirm the overlap detector actually fires on a known crosstalk clip.
python -c "
from pipeline import osd
out = osd('crosstalk_sample.wav')
print('overlap regions:', len(list(out.get_timeline().support())))
"
# expected: >= 1 region on a clip with a real interruption

# 2. Confirm dual labels are written where speakers collide.
python -c "
from pipeline import enrich_with_overlap
ann = enrich_with_overlap('crosstalk_sample.wav')
import collections
c = collections.Counter(len({l for _,_,l in ann.crop(seg).itertracks(yield_label=True)})
                        for seg in ann.get_timeline().support())
print('regions by concurrent-speaker count:', dict(c))
"
# expected: at least one region with 2 concurrent speakers

Then measure the improvement. Overlap-aware diarization error rate is the same metric as ordinary DER but with skip_overlap=False, so the seconds where two people talk are actually scored instead of silently excluded.

# pyannote.metrics==3.2.1
from pyannote.metrics.diarization import DiarizationErrorRate
from pyannote.database.util import load_rttm

reference = load_rttm("ep131_reference.rttm")["ep131"]
hypothesis = enrich_with_overlap("ep131.wav")

# skip_overlap=False makes overlap regions count toward the score.
metric = DiarizationErrorRate(collar=0.25, skip_overlap=False)
der = metric(reference, hypothesis, detailed=True)
print(f"overlap-aware DER: {der['diarization error rate']:.3f}")
print(f"missed detection : {der['missed detection']:.3f}")

A healthy result shows the overlap-aware DER dropping materially against the base pipeline (the interruptions that were pure missed-detection now score as correct), while the non-overlap DER is unchanged. If overlap-aware DER falls but non-overlap DER rises, the reconciler is over-attributing — tighten min_duration_on.

Failure Modes & Edge Cases

Edge case Symptom Remediation
min_duration_on too low Plosives and breath flagged as overlap, spurious dual labels Raise toward 0.25; verify against a clean monologue that should yield zero regions
Overlap region with only one diarized speaker Reconciler logs “<2 speakers”, leaves single label Widen the crop window slightly, or lower diarization clustering threshold so the quiet interrupter clusters
Three-plus concurrent speakers (panel) Only top two attributed, third voice dropped Take top_two → top_n; expect DER to stay high, panels are inherently hard
Word timestamps drift vs diarization frames Overlap labels land on adjacent words Align first via timestamp correction; never reconcile on unaligned words
skip_overlap=True left in eval DER looks great but ignores the region you fixed Always score with skip_overlap=False when reporting overlap gains
Laughter / applause under speech Detector fires, but there is no real second speaker Gate overlap regions on a VAD speech-probability floor before attributing
Very short interruption (“mm-hmm”) Region below min_duration_on, no label Lower the floor only for backchannel-heavy feeds; accept some false positives

For conversational feeds where crosstalk is constant, treat overlap attribution as a first-class stage rather than a cleanup pass, and keep the Speaker Diarization with Pyannote base configuration unchanged so your non-overlap accuracy never regresses while you tune the detector.

FAQ

Doesn't speaker-diarization-3.1 already handle overlap on its own?

Its segmentation head models overlap internally, but the pipeline resolves clusters into a turn timeline that under-reports genuine concurrency, so the interrupting speaker's frames often never reach the output annotation. Running an explicit overlapped-speech detector on segmentation-3.0 and promoting the top-two active speakers recovers those regions without touching the clustering that makes the base pipeline accurate on clean speech.

Why does my overlap-aware DER look worse than the number I reported before?

Because the earlier number almost certainly used skip_overlap=True, which excludes overlap regions from scoring entirely — you were grading only the easy frames. Setting skip_overlap=False counts the hard seconds, so the baseline rises and the honest improvement from attribution becomes visible against it. Report both figures with the flag stated explicitly.

How do I attribute overlap when three or more people talk at once?

The detector still flags the region, but the reconciler as written keeps only the two most-covering speakers. Extend the top_two selection to top_n and write every speaker whose in-region coverage exceeds a floor. Expect diarization error to stay elevated on dense panel crosstalk regardless — three concurrent voices are near the ceiling of what frame-level attribution can resolve reliably.