Reducing Hallucinations in AssemblyAI Outputs

Silencing phantom transcript text on dead air is what this page addresses, sitting beneath the Whisper Large V3 Integration Guide within the broader Transcription & Speaker Diarization pipeline. It solves one specific scenario: a production AssemblyAI integration that returns clean transcripts on studio audio but invents phantom phrases, repetition loops, and filler the moment a podcast or video clip hits extended silence, room tone, or a low-energy remote guest mic.

Problem Framing

Hallucination in a hosted transcription engine is not random noise — it is the decoder’s autoregressive priors filling an acoustic vacuum. When a frame carries little phonetic evidence (sub-20 dB SNR, or silence past roughly 800 milliseconds), the model stops transcribing what it hears and starts predicting what usually comes next. In media workflows this shows up as confident, fully-punctuated sentences emitted over dead air. Your validation log fills with telltale entries:

WARN  confidence  segment=ep88_00:21:04  text="thanks for watching don't forget to subscribe"  avg_conf=0.41
WARN  repetition  segment=ep88_00:37:55  pattern="you know you know you know"  repeats=6
WARN  hallucination  segment=ep88_00:52:18  reason="912ms silence, decoder emitted 9 tokens with no acoustic support"

Those “thanks for watching” and “don’t forget to subscribe” strings are a classic tell: they are high-frequency priors from the training distribution, surfacing exactly where the audio gave the model nothing. Left unchecked, a single phantom sentence propagates into show notes, search indexing, chapter titles, and — worst of all — corrupts speaker labels when the hallucinated tokens land on a frame that downstream speaker diarization with Pyannote then attributes to the wrong person. Turning down a temperature knob does not fix this, because you do not own the decoder. The fix is to stop feeding the model frames that have no speech in them, constrain its generative latitude through the API, and gate its output on confidence before anything is published.

Solution Architecture

The remedy has three deterministic stages and none of them require model access. First, a Voice Activity Detection (VAD) pre-filter slices the audio into active-speech chunks and discards non-speech frames before a single byte reaches AssemblyAI — this removes the silence that triggers the priors in the first place. Second, the submission payload pins the API parameters that otherwise let the decoder invent syntactic structure. Third, a post-transcription pass cross-references word-level confidence against a strict baseline and flags any segment that breaches it for fallback review rather than publishing a guess.

Three-stage hallucination-suppression flow for AssemblyAI Stage one runs raw episode audio through a Silero VAD gate at speech-probability threshold 0.65, dropping non-speech frames and merging gaps under 500 milliseconds into active-speech segments. Stage two submits those segments with a deterministic payload — punctuate and disfluencies disabled, speech_threshold 0.5 — to the AssemblyAI engine, returning a transcript with per-word confidence. Stage three feeds the transcript to a confidence validator: if more than 15 percent of words fall below 0.75 confidence the segment is flagged and routed to fallback review and diarization, otherwise it is published to the CMS. 1 2 3 STAGE 1 · VAD PRE-FILTER STAGE 2 · DETERMINISTIC SUBMISSION STAGE 3 · CONFIDENCE VALIDATION gated audio validated transcript + confidence Raw episode audio podcast · field · remote mic Silero VAD gate threshold 0.65 · drop non-speech Active-speech segments silence cut · gaps <500ms merged Deterministic payload punctuate · disfluencies = false AssemblyAI engine speech_threshold = 0.5 Transcript returned with per-word confidence Confidence validator per-word conf vs 0.75 >15% of words below 0.75? clean flagged Publish to CMS show notes · chapters · index Flag for fallback review Pyannote diarization · second engine

Because VAD runs locally and cheaply, you can route by source quality: high-fidelity studio recordings bypass the pre-filter entirely, while compressed field recordings and remote-guest exports trigger strict acoustic gating. That keeps the engine in the parent integration guide untouched while specializing the ingest path per feed.

Implementation

1. Pre-filter audio with VAD

Raw audio with sub-20 dB SNR or silence past 800 ms consistently triggers hallucination loops, so isolate active speech first. The following pass uses Silero VAD (loaded via torch.hub) to extract speech segments, merge micro-gaps under 500 ms to preserve contextual continuity, and abort cleanly on an all-silence file so you never submit an empty payload.

# torch==2.3.1  pydub==0.25.1  numpy==1.26.4  (silero-vad pulled via torch.hub)
import logging
import numpy as np
import torch
from pydub import AudioSegment
from typing import List, Tuple

logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
logger = logging.getLogger(__name__)

# Load Silero VAD via torch.hub (requires internet access on first call;
# cache the model for production use to avoid latency spikes).
vad_model, vad_utils = torch.hub.load(
    repo_or_dir="snakers4/silero-vad",
    model="silero_vad",
    force_reload=False,
    onnx=False
)
get_speech_timestamps = vad_utils[0]

def filter_hallucination_triggers(
    audio_path: str,
    vad_threshold: float = 0.65,    # speech probability cutoff; higher = stricter
    min_gap_merge_ms: int = 500     # gaps below this are bridged to keep context
) -> List[Tuple[int, int]]:
    """
    Pre-filters audio using Silero VAD to isolate high-probability speech segments.
    Returns a list of (start_ms, end_ms) tuples for active speech.
    """
    try:
        wav = AudioSegment.from_file(audio_path).set_frame_rate(16000).set_channels(1)
        audio_np = np.array(wav.get_array_of_samples(), dtype=np.float32) / 32768.0
        audio_tensor = torch.from_numpy(audio_np)

        logger.info("Running VAD inference on normalized 16kHz mono audio...")
        # return_seconds=True returns float timestamps; multiply by 1000 for ms.
        timestamps = get_speech_timestamps(
            audio_tensor,
            vad_model,
            threshold=vad_threshold,
            min_speech_duration_ms=300,   # discard sub-300ms blips (clicks, breaths)
            max_speech_duration_s=30.0,    # cap chunk length for API limits
            sampling_rate=16000,
            return_seconds=True
        )

        if not timestamps:
            logger.warning("No speech segments detected. Aborting to prevent empty payload submission.")
            return []

        # Merge adjacent gaps < min_gap_merge_ms to prevent fragmentation-induced context loss
        active_segments = []
        current_start_ms = int(timestamps[0]['start'] * 1000)
        current_end_ms = int(timestamps[0]['end'] * 1000)

        for seg in timestamps[1:]:
            seg_start_ms = int(seg['start'] * 1000)
            seg_end_ms = int(seg['end'] * 1000)
            if seg_start_ms - current_end_ms < min_gap_merge_ms:
                current_end_ms = seg_end_ms
            else:
                active_segments.append((current_start_ms, current_end_ms))
                current_start_ms, current_end_ms = seg_start_ms, seg_end_ms
        active_segments.append((current_start_ms, current_end_ms))

        logger.info(f"VAD gating complete. Extracted {len(active_segments)} speech segments.")
        return active_segments

    except FileNotFoundError as e:
        logger.error(f"Audio file not found: {audio_path} | {e}")
        raise
    except Exception as e:
        logger.error(f"VAD pipeline failure: {e}")
        raise RuntimeError("Acoustic pre-filtering failed. Check audio format and VAD dependencies.")

The vad_threshold of 0.65 is the load-bearing parameter: lower it and silence leaks through, raise it and you start clipping soft-spoken speech. Tune it per source profile, not globally.

2. Submit with deterministic API parameters

Once the audio is segmented, the AssemblyAI payload must suppress generative overconfidence. The default config leaves punctuate and disfluencies enabled, which compounds hallucination by forcing the decoder to invent syntactic structure where none exists. For technical and interview formats, disable disfluencies, set punctuate to False if downstream NLP handles punctuation, enable language_detection to prevent cross-lingual phantom generation, and set speech_threshold to 0.5 to drop low-confidence word tokens before they reach the diarization layer. The official AssemblyAI API documentation carries the current parameter schema and rate limits.

# Deterministic submission payload — every generative latitude is pinned off.
ASSEMBLYAI_CONFIG = {
    "audio_url": "https://storage.example.com/segment_01.wav",
    "punctuate": False,           # let downstream NLP punctuate; don't invent syntax
    "disfluencies": False,        # stop the decoder fabricating "um/uh" filler
    "language_detection": True,   # block cross-lingual phantom tokens
    "speech_threshold": 0.5,      # drop word tokens below this confidence
    "auto_chapters": False,       # never chapter un-validated text
    "word_boost": ["technical", "domain-specific", "brand-terms"],
    "dual_channel": False
}

3. Validate word-level confidence before ingestion

Raw API responses must be filtered deterministically before they reach any content management system. Cross-reference returned word-level confidence against a strict baseline — here, more than 15% of words below 0.75 confidence flags the whole segment for fallback review instead of publication.

def validate_transcript_payload(response_json: dict, min_confidence: float = 0.75) -> dict:
    """Flag segments where hallucinated low-confidence tokens exceed 15% of words."""
    words = response_json.get("words", [])
    low_conf_words = [w for w in words if w.get("confidence", 0) < min_confidence]
    if words and len(low_conf_words) > len(words) * 0.15:
        logger.warning(
            f"High hallucination risk: {len(low_conf_words)} of {len(words)} words "
            f"below {min_confidence} confidence."
        )
        return {"status": "flagged", "low_confidence_count": len(low_conf_words), "data": response_json}
    return {"status": "clean", "data": response_json}

When AssemblyAI’s native speaker separation struggles with overlapping dialogue or rapid turn-taking, route the VAD-segmented audio to speaker diarization with Pyannote for frame-accurate attribution, and reconcile any drifting word timestamps through the timestamp alignment and correction workflow before the transcript reaches chapter generation. In production, decouple all of this from synchronous request cycles by polling and retrying through the async transcription queue management layer rather than blocking a web request on a multi-minute job.

Verification

Confirm each stage independently before you trust the pipeline on a full back catalogue.

# 1. Prove VAD is actually dropping silence — inspect SNR and the gated segment count.
#    ffprobe the source, then compare runtime before/after gating.
ffprobe -v error -show_entries format=duration -of default=noprint_wrappers=1:nokey=1 source.wav

# 2. Verify no empty payloads escape: an all-silence clip must yield zero segments.
python -c "from pipeline import filter_hallucination_triggers as f; print(f('silence_test.wav'))"
# expected: WARN 'No speech segments detected...'  ->  []

# 3. Replay a known-bad episode and assert the flagged-segment count drops.
python -c "
from pipeline import validate_transcript_payload
import json
resp = json.load(open('ep88_response.json'))
print(validate_transcript_payload(resp)['status'])
"
# expected: 'clean' on the VAD-gated re-run (was 'flagged' before gating)

A healthy run shows the silence-only test returning an empty segment list (no submission), the gated re-transcription of a previously corrupted episode coming back clean, and your hallucination counter at near zero on held-out low-energy frames. Export VAD rejection rates, API confidence histograms, and flag frequencies to your observability stack (Prometheus, Datadog, or ELK) so you can keep tuning vad_threshold and min_confidence against real drift.

Failure Modes & Edge Cases

Edge case Symptom Remediation
vad_threshold set too high Soft-spoken or whispered speech clipped out, transcript gaps Lower to 0.5–0.6 for that source profile; verify against a quiet-guest sample
vad_threshold set too low Room tone survives the gate, phantom “subscribe” strings persist Raise toward 0.7; add a min_speech_duration_ms floor to kill blips
Over-aggressive gap merge Two speakers bridged into one segment, diarization mislabels Reduce min_gap_merge_ms; cut hard at known speaker-change points
language_detection flips mid-episode Bilingual guest triggers cross-lingual tokens Pin a fixed language_code per known feed instead of auto-detect
Confidence gate too strict (> 0.8) Clean accented speech flagged as risk, fallback cost spikes Lower min_confidence to 0.7; widen the 15% segment tolerance
Repetition loop survives gating "you know you know..." over real low-SNR speech Add an n-gram repetition detector post-transcription; re-queue through DeepFilterNet
Studio audio over-processed VAD adds latency with no quality gain Route high-SNR sources past the pre-filter; gate only field/remote exports

For cost control, classify incoming media at ingest: treat AssemblyAI as the primary acoustic parser for standard segments and reserve a fallback engine for segments that breach the confidence gate. The complementary local approach — adapting the model to your own lexicon so it stops substituting on domain terms — is documented in fine-tuning Whisper for technical podcasts.

FAQ

Why does AssemblyAI hallucinate "thanks for watching" over silence?

That phrase is a high-frequency prior from the model's training distribution. When a frame carries no phonetic evidence, the decoder's autoregressive head predicts the most probable continuation rather than transcribing the audio — and across web-scraped video data, sign-off boilerplate is extremely common. Removing the silent frames with VAD before submission is the only reliable fix, because you cannot edit a hosted decoder's priors.

Should I gate every file through VAD, or only some?

Only the ones that need it. High-SNR studio recordings rarely hallucinate, so routing them through VAD just adds latency. Classify by source at ingest and trigger the pre-filter for compressed field recordings, remote-guest exports, and anything with long silences. Keep the studio fast path direct to the API.

What confidence threshold should flag a segment for review?

Start at `0.75` per word with a 15% segment tolerance — if more than 15% of words in a segment fall below `0.75`, flag it. Tighten or loosen against your own confidence histograms: accented but accurate speakers can sit lower without being hallucinations, so a single global cutoff will over-flag. Tune per feed and watch your false-flag rate.