Fine-Tuning Whisper for Technical Podcasts

Adapting Whisper to a show’s own vocabulary is the work covered here, a page that sits beneath the Whisper Large V3 Integration Guide within the broader Transcription & Speaker Diarization pipeline. It solves one specific scenario: a stock Whisper checkpoint that posts an acceptable word error rate on conversational audio but collapses into invented terminology the moment a guest starts rattling off framework names, CLI flags, and acronyms on an engineering show.

Problem Framing

Stock Whisper Large V3 was trained on broad web audio, so it has weak priors for dense domain nomenclature. On a technical podcast you hit three compounding failure modes at once: domain-specific vocabulary the decoder has barely seen, rapid-fire acronym density, and inconsistent microphone quality across remote guests. The result is not random noise — it is confident substitution. The model replaces an unfamiliar token with the nearest phonetic neighbour from its priors, and your evaluation log fills with telltale entries:

WARN  wer.eval  segment=ep142_00:14:32  ref="deploy with kubectl apply"  hyp="deploy with cube cuddle apply"  wer=0.50
WARN  wer.eval  segment=ep142_00:31:07  ref="the gRPC interceptor"        hyp="the gee RPC inter-ceptor"   wer=0.40
WARN  hallucination  segment=ep142_00:48:11  reason="low-energy frame, decoder emitted 7 tokens with no acoustic support"

Whisper’s cross-attention mechanism is the proximate cause: when a frame carries little acoustic information — a low signal-to-noise guest mic, or trailing room tone — attention entropy climbs and the decoder leans on its language-model priors instead of the spectrogram. Across a single 90-minute episode this can push aggregate WER from a tolerable 8% into the high teens, and every misheard product name propagates downstream into show notes, search indexing, and chapter titles. Fixing it requires teaching the model your lexicon, not turning up the temperature knob.

Solution Architecture

The fix has three stages, and none of them touches Whisper’s base weights. First, curate a small, ruthlessly clean training set: VAD-sliced segments, gated on signal-to-noise ratio, with acronyms normalized to a deterministic spoken form. Second, attach Low-Rank Adaptation (LoRA) adapters to the attention projection layers so a single A10G can train the model without the cost of full fine-tuning. Third, route the resulting adapter back into the pipeline behind a confidence gate, falling back to a premium engine only for the segments that still breach threshold.

Three-stage Whisper fine-tuning flow: SNR-gated data curation, LoRA training, and confidence-gated serving Stage one curates data: raw remote-guest episode audio is VAD-sliced, passed through a signal-to-noise gate that drops any segment below 12 decibels, acronym-normalized, and written to a JSONL dataset of audio, text, and speaker_id fields. Stage two trains: the frozen Whisper Large V3 base with LoRA adapters on the q_proj and v_proj attention projections at rank 16 feeds a Seq2SeqTrainer running fp16 with an effective batch size of 32, evaluated each epoch against word error rate with early stopping where lower is better, looping until it emits a few-megabyte trained adapter. Stage three serves: the adapter is loaded per job behind a confidence gate; segments scoring at or above 0.65 publish to show notes, search, and chapters, while segments below 0.65 route on the accent branch to a DeepFilterNet noise-suppression and cloud-fallback re-queue. Stage 1 — Curate: ruthlessly clean, SNR-gated training segments Raw episode audio remote-guest WAVs VAD slice 0.5 speech · 0.3 s sil SNR gate keep ≥ 12 dB Acronym normalize K8s → Kubernetes JSONL dataset audio · text · speaker_id drop SNR < 12 dB Stage 2 — Train: parameter-efficient LoRA, never touching base weights Whisper Large V3 + LoRA base frozen · q_proj / v_proj r=16 · alpha=32 · <1% trainable Seq2SeqTrainer fp16 · eff. batch 32 WER eval early-stop · lower = better Trained adapter few MB · per show epoch loop feed training segments Stage 3 — Serve: load per job behind the pipeline confidence gate Adapter serving base resident · load per job Confidence gate score ≥ 0.65 ? Publish transcript show notes · search · chapter titles DeepFilterNet → cloud fallback noise-suppress · re-queue < 0.65 deploy adapter to pipeline

Because the adapter is tiny, you can keep several domain-specific adapters (one per show, or per topic vertical) and load the right one at inference time without re-deploying the base model. That keeps the engine described in the parent integration guide untouched while specializing its output per feed.

Implementation

1. Curate the dataset

Extract 30-second to 2-minute segments with a Voice Activity Detection pass set to a minimum speech threshold of 0.5 and a minimum silence duration of 0.3 seconds. Drop any segment whose signal-to-noise ratio falls below 12 dB — those are exactly the low-energy frames that trigger hallucination. Store the result as JSON Lines with audio, text, and speaker_id fields, and normalize acronyms to a deterministic spoken form so tokenization stays phonetically aligned. Maintain strict temporal boundaries so a segment never spans a speaker change, which would otherwise corrupt downstream speaker diarization with Pyannote.

# transformers==4.44.2  datasets==2.21.0  librosa==0.10.2  numpy==1.26.4
import json
import logging
import re
import librosa
import numpy as np
from datasets import Dataset

logging.basicConfig(level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s")
logger = logging.getLogger(__name__)

# Deterministic acronym normalization: expand to a spoken form so the
# tokenizer aligns phonetically instead of guessing letter-by-letter.
ACRONYM_MAP = {
    r"\bK8s\b": "Kubernetes",
    r"\bCI/CD\b": "C I slash C D",
    r"\bAPI\b": "A P I",
    r"\bML\b": "M L",
}

def normalize_text(text: str) -> str:
    for pattern, replacement in ACRONYM_MAP.items():
        text = re.sub(pattern, replacement, text)
    return text

def preprocess_dataset(jsonl_path: str, target_sr: int = 16000) -> Dataset:
    records = []
    try:
        with open(jsonl_path, "r", encoding="utf-8") as f:
            for line_num, line in enumerate(f, 1):
                try:
                    entry = json.loads(line)
                    audio_arr, sr = librosa.load(entry["audio"], sr=target_sr)
                    # Estimate SNR from the ratio of loud vs quiet RMS frames.
                    rms = librosa.feature.rms(y=audio_arr)[0]
                    signal_rms = np.percentile(rms, 90)
                    noise_rms = np.percentile(rms, 10) + 1e-8
                    snr_db = 20 * np.log10(signal_rms / noise_rms)
                    if snr_db < 12.0:  # below this, the decoder hallucinates
                        logger.warning(f"Line {line_num}: SNR {snr_db:.2f} dB below threshold. Skipping.")
                        continue

                    records.append({
                        "audio": {"array": audio_arr, "sampling_rate": target_sr},
                        "text": normalize_text(entry["text"]),
                        "speaker_id": entry.get("speaker_id", "unknown"),
                    })
                except Exception as e:
                    logger.error(f"Line {line_num} parsing failed: {e}")
                    continue
    except FileNotFoundError:
        logger.critical(f"Dataset file not found at {jsonl_path}")
        raise

    return Dataset.from_list(records)

2. Attach LoRA adapters and train

Full fine-tuning of Whisper Large V3 is prohibitive for most media teams, so target the attention projections with LoRA at rank r=16, alpha=32, and dropout=0.05. Drive the learning rate with a cosine schedule from 2e-4 and a 0.03 warmup ratio. Set gradient accumulation to 8 to simulate an effective batch size of 32 on a single 24 GB A10G without an out-of-memory kill on long sequences. The target_modules list matches q_proj and v_proj by substring, which covers both self-attention and cross-attention in the Hugging Face Whisper implementation.

# transformers==4.44.2  peft==0.12.0  evaluate==0.4.2  torch==2.3.1
from peft import LoraConfig, get_peft_model
from transformers import (
    WhisperForConditionalGeneration,
    WhisperProcessor,
    Seq2SeqTrainingArguments,
    Seq2SeqTrainer,
)

def configure_lora_model(model_name: str = "openai/whisper-large-v3"):
    model = WhisperForConditionalGeneration.from_pretrained(model_name)
    processor = WhisperProcessor.from_pretrained(model_name)

    lora_config = LoraConfig(
        r=16,                 # rank of the low-rank update matrices
        lora_alpha=32,        # scaling; effective LR ~ alpha/r
        target_modules=["q_proj", "v_proj"],  # attention query/value projections
        lora_dropout=0.05,
        bias="none",
        task_type="SEQ_2_SEQ_LM",
    )
    model = get_peft_model(model, lora_config)
    model.print_trainable_parameters()  # expect <1% of base params trainable
    return model, processor

def setup_trainer(model, processor, train_dataset, eval_dataset):
    training_args = Seq2SeqTrainingArguments(
        output_dir="./whisper-tech-podcast-finetuned",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,   # effective batch size 32
        learning_rate=2e-4,
        warmup_ratio=0.03,
        lr_scheduler_type="cosine",
        fp16=True,
        logging_steps=50,
        save_steps=500,
        eval_strategy="steps",
        eval_steps=500,
        save_total_limit=3,
        load_best_model_at_end=True,
        metric_for_best_model="wer",
        greater_is_better=False,         # lower WER is better
    )

    import evaluate
    wer_metric = evaluate.load("wer")

    def compute_metrics(pred):
        pred_ids = pred.predictions
        label_ids = pred.label_ids
        label_ids[label_ids == -100] = processor.tokenizer.pad_token_id
        pred_str = processor.batch_decode(pred_ids, skip_special_tokens=True)
        label_str = processor.batch_decode(label_ids, skip_special_tokens=True)
        return {"wer": wer_metric.compute(predictions=pred_str, references=label_str)}

    return Seq2SeqTrainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        tokenizer=processor.feature_extractor,
        compute_metrics=compute_metrics,
    )

Once trained, the adapter weights are a few megabytes. Serve them behind the same confidence gate the pipeline already uses: when an inference confidence score falls below 0.65, pass the segment through a noise-suppression preprocessor (RNNoise or DeepFilterNet) and re-queue it through the async transcription queue management layer rather than publishing a guess.

Verification

Confirm each stage independently before you trust the adapter in production.

# 1. Adapter is actually small — trainable params should be well under 1% of base.
#    print_trainable_parameters() prints e.g.:
#    trainable params: 1,966,080 || all params: 1,545,271,360 || trainable%: 0.1272

# 2. Score the merged model against a held-out set of 500+ technical segments.
python -c "
import evaluate
wer = evaluate.load('wer')
# refs/hyps loaded from your eval run
print('WER', wer.compute(predictions=hyps, references=refs))
"

# 3. Watch VRAM during a training step to prove grad-accumulation isn't fragmenting memory.
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 5

A healthy run shows validation WER dropping below the conversational baseline on the technical eval set within three epochs, nvidia-smi holding steady under the 24 GB ceiling, and the hallucination counter at near zero on held-out low-energy frames. If Whisper’s native timestamps still drift by ±200 ms on rapid dialogue, reconcile them through the timestamp alignment and correction workflow before the transcript reaches show-note generation.

Failure Modes & Edge Cases

Edge case Symptom Remediation
Validation WER plateaus above 18% after epoch 3 Loss flat, WER stuck Lower lora_dropout to 0.01; confirm the corpus covers your target lexicon phonetically
VRAM fragmentation under grad accumulation CUDA out of memory mid-epoch despite headroom Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True; lower per_device_train_batch_size, raise accumulation to hold batch 32
Over-aggressive acronym map Real words rewritten (e.g. a name containing “ML”) Make ACRONYM_MAP patterns word-boundary and case-exact; keep the map per-show
Speaker bleed across segment boundary Adapter learns crosstalk as one speaker Enforce hard temporal cuts at diarization change points before slicing
Low-SNR guest mic survives the gate Hallucinated tokens on a specific guest Re-run that guest’s segments through DeepFilterNet, then re-queue below the 0.65 confidence gate
Catastrophic forgetting on general audio Fine-tuned model worse on casual talk Keep adapters per-domain and load conditionally; never merge a domain adapter into the shared base

For cost control, classify incoming media at ingest: route standard conversational segments to the fine-tuned local adapter and reserve a premium cloud engine for heavily accented or highly specialized segments that need human review. The reducing hallucinations in AssemblyAI outputs page documents the complementary confidence-gating heuristics for that fallback path.

FAQ

How many hours of audio do I need to fine-tune effectively?

For a domain adapter, 5–15 hours of tightly curated, SNR-gated segments usually moves the needle more than 100 hours of noisy raw audio. LoRA updates a fraction of a percent of the parameters, so it specializes the lexicon quickly and overfits fast — quality and label accuracy of the training set dominate over raw quantity.

Should I expand acronyms in the transcript text or leave them literal?

Normalize them to a deterministic spoken form during training so tokenization aligns with how the words are actually pronounced, then reverse the mapping in a deterministic post-processing pass for display. Training on the literal "K8s" string forces the decoder to guess letter spellings it rarely heard, which is precisely what produces phonetic substitutions.

Can I serve several show-specific adapters from one base model?

Yes — that is the main reason to use LoRA here. Keep the base Whisper Large V3 weights resident in VRAM and load the relevant few-megabyte adapter per job. Never merge a single domain adapter back into the shared base, or you trade general-audio accuracy for one show's gains.