Golden-File Testing for Transcode Output

This page sits beneath CI/CD and testing for media pipelines in the wider Pipeline Automation & Batch Processing layer, and it addresses the case where you want to catch any unintended change in a transcode — a shifted bitrate ladder, a subtly different picture, a re-timed audio track — but the output is never byte-for-byte identical, so a naïve file-diff test fails on every run.

Problem Framing

Transcoding is not deterministic at the byte level. The x264 and x265 encoders embed build identifiers and make thread-scheduling-dependent decisions; the MP4 muxer writes timestamps and padding that vary; even re-running the identical command on the same machine can produce files whose bytes differ while the decoded picture and sound are perceptually identical. So the obvious test is worthless:

$ md5sum candidate.mp4 golden.mp4
0f2e...a1   candidate.mp4
9b74...c3   golden.mp4          # different every run — this test is pure noise

Yet the risk it was trying to guard is real. A refactor of the encode ladder, a change from -preset medium to -preset fast, or a filter reordering can degrade quality or shift loudness without changing any field a codec contract pins. You need a test that answers “is this output meaningfully the same as the reference we blessed?” and tolerates the noise floor of encoder non-determinism while still failing on a real regression. That means comparing decoded signals and structured metadata against thresholds — not comparing bytes.

This is deliberately distinct from a contract test. A contract test asserts fixed properties from a spec (must be 48 kHz AAC in an MP4) with no reference file; the integration testing codec contracts workflow covers that. Golden-file testing asserts similarity to a known-good artifact, catching drift in properties no contract enumerates.

Solution Architecture

The comparison runs on three independent channels, each with its own tolerance, and the test fails if any channel breaches. Structured metadata (from ffprobe) is compared for exact equality on fields that must not move. The video signal is compared frame-against-frame with FFmpeg’s own PSNR and SSIM filters, gated on thresholds. The audio is decoded to raw PCM and compared by integrated loudness and a quantized hash, tolerating sample-level dither while catching a re-timed or re-gained track.

Splitting the comparison this way means a failure is diagnosable: a metadata mismatch points at a muxing or ladder change, a PSNR/SSIM drop points at the picture, and an audio breach points at loudness or timing — you know which layer moved before you open the file.

Implementation

1. Store the golden and a metadata sidecar

Commit the reference output alongside a JSON sidecar of its pinned metadata, so metadata comparison never has to decode the golden at test time and the expected values are reviewable in the diff.

# python 3.10+   numpy==1.26.4   soundfile==0.12.1
import json, subprocess, pathlib

def probe(path: str) -> dict:
    out = subprocess.run(
        ["ffprobe", "-v", "error", "-show_streams", "-show_format",
         "-of", "json", path], capture_output=True, text=True, check=True,
    ).stdout
    return json.loads(out)

def golden_fields(path: str) -> dict:
    m = probe(path)
    v = next(s for s in m["streams"] if s["codec_type"] == "video")
    a = next(s for s in m["streams"] if s["codec_type"] == "audio")
    return {
        "video": {"codec": v["codec_name"], "pix_fmt": v["pix_fmt"],
                   "width": v["width"], "height": v["height"],
                   "avg_frame_rate": v["avg_frame_rate"]},
        "audio": {"codec": a["codec_name"], "sample_rate": a["sample_rate"],
                   "channels": a["channels"]},
        "duration": round(float(m["format"]["duration"]), 2),
    }

2. Compare structured metadata for exact fields

These fields must not drift, so compare them for equality. This is the cheap first channel and it runs without decoding a frame.

def test_metadata_matches_golden():
    candidate = golden_fields("candidate.mp4")
    expected = json.loads(pathlib.Path("tests/golden/720p.json").read_text())
    assert candidate == expected, f"metadata drift: {candidate} != {expected}"

3. Compare the video signal with PSNR and SSIM

Run the candidate against the golden through FFmpeg’s psnr and ssim filters — FFmpeg does the frame alignment and math, printing a per-run average you parse and threshold. The filters are documented in the FFmpeg filters documentation.

import re

def compare_video(candidate: str, golden: str) -> tuple[float, float]:
    proc = subprocess.run(
        ["ffmpeg", "-hide_banner", "-i", candidate, "-i", golden,
         "-lavfi", "[0:v][1:v]psnr;[0:v][1:v]ssim", "-f", "null", "-"],
        capture_output=True, text=True,
    )
    log = proc.stderr
    psnr = float(re.search(r"average:([0-9.]+)", log).group(1))
    ssim = float(re.search(r"All:([0-9.]+)", log).group(1))
    return psnr, ssim

def test_video_within_quality_threshold():
    psnr, ssim = compare_video("candidate.mp4", "tests/golden/720p.mp4")
    assert psnr >= 42.0, f"PSNR {psnr} dB below 42 — picture regressed"
    assert ssim >= 0.98, f"SSIM {ssim} below 0.98 — structural change"

A PSNR of 42 dB and SSIM of 0.98 sit above the encoder’s non-determinism floor — repeated correct runs comfortably clear them — while a real preset or filter regression drops well below. Raise the floors for archival masters, lower them for lossy proxy renditions.

4. Compare audio by decoded PCM

Byte-comparing AAC is noise, so decode both tracks to raw PCM and compare on two axes: integrated loudness (catches a gain or loudnorm change) and a quantized hash (catches a re-timed or truncated track while tolerating dither).

import numpy as np, soundfile as sf, hashlib, io

def decode_pcm(path: str) -> np.ndarray:
    raw = subprocess.run(
        ["ffmpeg", "-v", "error", "-i", path, "-f", "wav", "-ac", "2",
         "-ar", "48000", "pipe:1"], capture_output=True, check=True,
    ).stdout
    data, _ = sf.read(io.BytesIO(raw), dtype="int16")
    return data

def quantized_hash(pcm: np.ndarray) -> str:
    # quantize to 8-bit to absorb sub-LSB dither, then hash the shape+values
    q = (pcm >> 8).astype(np.int8)
    return hashlib.sha256(q.tobytes()).hexdigest()

def test_audio_matches_golden():
    cand, gold = decode_pcm("candidate.mp4"), decode_pcm("tests/golden/720p.mp4")
    assert cand.shape == gold.shape, "audio length/channel drift"
    assert quantized_hash(cand) == quantized_hash(gold), "audio content changed"

Verification

Confirm the golden tests pass on a correct build, then prove each channel fails on a matching regression so you know the thresholds are not slack.

# 1. Green across all three channels on the correct pipeline
pytest tests/golden -q
# expected: 3 passed

# 2. Prove the video channel bites — re-encode the candidate at a worse preset
ffmpeg -y -i src.mp4 -c:v libx264 -preset ultrafast -crf 34 candidate.mp4
pytest tests/golden/test_video_within_quality_threshold.py -q
# expected: FAILED ... PSNR 31.2 dB below 42

# 3. Prove the audio channel bites — nudge the gain by 3 dB
ffmpeg -y -i src.mp4 -af volume=3dB -c:v copy candidate.mp4
pytest tests/golden/test_audio_matches_golden.py -q
# expected: FAILED ... audio content changed

A healthy suite is three green channels on the blessed pipeline and a clean, well-labeled failure on each injected regression. If a deliberately worse encode still passes the PSNR gate, your threshold is too low — tighten it until the bad build fails but repeated good builds stay green.

Failure Modes & Edge Cases

Fields that must match a fixed specification regardless of any reference — the mandated codec, container brand, and sample rate — belong in a contract test rather than a golden; pair this workflow with integration testing codec contracts so spec violations and reference drift are caught by the layer suited to each.

FAQ

Related

• Up to the parent guide: CI/CD and testing for media pipelines
• Sibling technique on fixed specs: integration testing codec contracts
• The loudness reference these comparisons track: audio codec normalization workflows
• Pipeline overview: Pipeline Automation & Batch Processing