Detecting Truncated Uploads with ffprobe

This page sits beneath media validation and error routing within the broader Media Ingestion & Format Architecture pipeline, and it solves one precise scenario: an upload that lands looking valid — right extension, parseable header — but whose byte stream was cut off mid-transfer, so the last minutes of audio or video simply do not exist. The goal is to catch it at the ingestion boundary with a cheap ffprobe pass, before a GPU is ever allocated to it.

Problem Framing

A truncated upload is more dangerous than a corrupt one, because it often passes a superficial check. A dropped connection during an S3 multipart upload, an interrupted SFTP transfer, or a browser tab closed mid-post leaves a file whose container header advertises a full-length asset while the media data behind it is short. Downstream, the symptoms are expensive and late:

ffprobe ep204.mp4: [mov,mp4] moov atom not found
ffprobe ep173.mp4: [aac] Estimating duration from bitrate, this may be inaccurate
transcode-worker: End of file reached at 41:12 of declared 58:30 — output truncated
diarization: speaker timeline ends at 2472s but transcript references 3510s (drift)

Two distinct failure shapes hide here. The first is the header-side break: the moov atom that indexes an MP4 was never written because the writer never reached the end, so ffprobe can’t parse the file at all (moov atom not found). The second, subtler shape is the body-side break: the moov is present (the file was faststart-remuxed, or the format stores its index up front) and declares 58 minutes, but only 41 minutes of mdat samples actually arrived. That second case decodes and transcodes happily right up to the cliff, then silently produces a short asset whose timestamps no longer line up with the transcript. Catching both means not trusting the declared duration alone.

Solution Architecture

The check is a two-read ffprobe sequence plus a size-versus-bitrate sanity guard, feeding a single verdict that the media validation and error routing layer acts on. The first read is cheap: pull format metadata (declared duration, size, bitrate) and catch a hard parse failure such as a missing moov. The second read is the honest one: count actual decoded packets to derive a real duration that owes nothing to the header. If the declared and decoded durations disagree beyond a tolerance, or the file size is impossibly small for its declared bitrate and duration, the asset is truncated and goes straight to quarantine — never to the compute broker.

Truncated-upload validation decision flow An uploaded asset enters a first ffprobe read of format metadata. If the container fails to parse, for example a missing moov atom, it is routed to quarantine. Otherwise a second ffprobe read counts decoded packets to compute a true duration. That true duration is compared with the declared duration and a size-versus-bitrate sanity check. If they agree within tolerance the asset is promoted to the compute broker; if they disagree the asset is routed to quarantine with a truncation error code. Upload ffprobe: format read ffprobe: count packets declared ≈ decoded? + size vs bitrate sanity tolerance ±2% or 2s whichever is larger Compute broker READY_FOR_TRANSCODE Quarantine / DLQ ERR_TRUNCATED_UPLOAD

The design rule mirrors the parent stage: validation reads headers and counts packets without decoding video frames, so it runs on cheap, high-memory workers and a fundamentally short file never reaches a GPU. This is the same separation of validation from transformation that keeps the whole media validation and error routing topology inexpensive.

Implementation

Pin ffprobe to the same FFmpeg 6.x build the rest of ingestion uses, and bound every probe so a pathological file can’t hang the worker. The function below runs the two reads and the sanity guard, returning a structured verdict.

# python 3.10+
# ffprobe from ffmpeg 6.x; no third-party deps
import json
import subprocess
from dataclasses import dataclass

PROBE_TIMEOUT = 60          # hard ceiling; a truncated file must fail fast
DURATION_TOL_S = 2.0        # absolute floor for the tolerance
DURATION_TOL_FRAC = 0.02    # 2% of declared duration

@dataclass(frozen=True)
class TruncationVerdict:
    ok: bool
    declared_s: float | None
    decoded_s: float | None
    error_code: str | None
    detail: str

def _run(cmd: list[str]) -> subprocess.CompletedProcess:
    return subprocess.run(cmd, capture_output=True, text=True, timeout=PROBE_TIMEOUT)

def check_truncation(path: str) -> TruncationVerdict:
    # Read 1 — format metadata. A parse failure here (e.g. missing moov) is terminal.
    fmt = _run(["ffprobe", "-v", "error", "-show_entries",
                "format=duration,size,bit_rate", "-of", "json", path])
    if fmt.returncode != 0 or "moov atom not found" in fmt.stderr:
        return TruncationVerdict(False, None, None, "ERR_TRUNCATED_UPLOAD",
                                 f"container unreadable: {fmt.stderr.strip()[:120]}")
    meta = json.loads(fmt.stdout or "{}").get("format", {})
    declared = float(meta.get("duration", 0.0) or 0.0)
    size = int(meta.get("size", 0) or 0)
    bitrate = int(meta.get("bit_rate", 0) or 0)

    # Size-vs-bitrate sanity: a 58-min file at 2 Mbps can't be 40 MB.
    if bitrate and declared:
        expected_bytes = bitrate / 8 * declared
        if size < expected_bytes * 0.5:
            return TruncationVerdict(False, declared, None, "ERR_TRUNCATED_UPLOAD",
                                     f"size {size}B is <50% of expected {int(expected_bytes)}B")

    # Read 2 — decode-count the true duration from packet PTS/duration, ignoring the header.
    pkt = _run(["ffprobe", "-v", "error", "-select_streams", "a:0",
                "-show_entries", "packet=pts_time,duration_time",
                "-of", "csv=p=0", path])
    times = [ln.split(",") for ln in pkt.stdout.splitlines() if "," in ln]
    decoded = 0.0
    if times:
        last_pts = max((float(t[0]) for t in times if t[0]), default=0.0)
        last_dur = next((float(t[1]) for t in reversed(times) if len(t) > 1 and t[1]), 0.0)
        decoded = last_pts + last_dur

    tol = max(DURATION_TOL_S, declared * DURATION_TOL_FRAC)
    if declared and abs(declared - decoded) > tol:
        return TruncationVerdict(False, declared, decoded, "ERR_TRUNCATED_UPLOAD",
                                 f"declared {declared:.1f}s vs decoded {decoded:.1f}s (tol {tol:.1f}s)")
    return TruncationVerdict(True, declared, decoded, None, "duration within tolerance")

The key move is the second read. ffprobe -show_entries packet=pts_time,duration_time walks the actual demuxed packets, so the duration it yields comes from bytes that really exist. When the header claims 58 minutes but only 41 minutes of packets are present, decoded lands at ~2472s while declared reads ~3510s, blowing past the tolerance and producing ERR_TRUNCATED_UPLOAD. The size-versus-bitrate guard is a cheap short-circuit that catches gross truncation before the packet walk even runs.

Verification

Confirm the check on three fixtures: a whole file, a header-broken file, and a body-truncated file.

# Build a known-good and two broken fixtures.
ffmpeg -f lavfi -i sine=d=60 -c:a aac good.m4a
head -c 400000 good.m4a > body_truncated.m4a          # cut the mdat mid-stream
ffmpeg -f lavfi -i sine=d=60 -c:a aac -movflags '' nofaststart.mp4  # moov at tail

# 1. The good file must pass.
python -c "from validate import check_truncation as c; print(c('good.m4a'))"
# expected: TruncationVerdict(ok=True, ... 'duration within tolerance')

# 2. The body-truncated file must fail on the declared-vs-decoded gap.
python -c "from validate import check_truncation as c; print(c('body_truncated.m4a'))"
# expected: ok=False, error_code='ERR_TRUNCATED_UPLOAD', decoded << declared

# 3. Directly confirm the two durations disagree with raw ffprobe.
ffprobe -v error -show_entries format=duration -of csv=p=0 body_truncated.m4a  # declared
ffprobe -v error -select_streams a:0 -show_entries packet=pts_time \
  -of csv=p=0 body_truncated.m4a | tail -1                                     # last real packet

A correct implementation passes the whole file, fails both broken fixtures with ERR_TRUNCATED_UPLOAD, and never spends more than PROBE_TIMEOUT seconds on any input. Wire the verdict’s error_code and detail into the structured error payload so the quarantine record explains exactly why the asset was rejected.

Failure Modes & Edge Cases

Edge case Symptom Remediation
VBR audio, header estimates duration Declared duration is fuzzy; false positives near tolerance Widen tolerance for VBR, or trust the decoded packet count only
Legitimate faststart with tail padding Tiny declared/decoded gap on a valid file Keep tolerance at max(2s, 2%); don’t set it tighter than the container’s rounding
Live-streamed / fragmented MP4 No single declared duration in format Sum fragment durations, or validate after the fragments are finalized
moov present but mdat short File decodes then cuts off silently The declared-vs-decoded read catches it; do not rely on parse success alone
Zero-length or empty upload format read returns no duration Treat missing/zero declared duration as ERR_TRUNCATED_UPLOAD
Unbounded probe on a huge file Worker stalls scanning Keep PROBE_TIMEOUT and consider -probesize/-analyzeduration caps

For files that fail the parse entirely (a missing moov from an interrupted writer), the remediation may be a faststart-style repair rather than outright rejection — the recovery paths live in handling corrupt MP4 files in automated pipelines, and the quarantine-versus-fallback decision itself is routed by setting up fallback routing for failed transcodes.

FAQ

Why not just trust the duration in the container header?

Because a truncated file's header often still claims the original full length — the writer recorded the intended duration before the transfer was cut off. The only trustworthy duration comes from counting the packets that actually arrived, which is what the second ffprobe read does. Comparing that against the declared value is what surfaces a body-side truncation.

Is this check expensive enough to worry about at scale?

No. Both reads are metadata/packet walks, not frame decodes, so they run on cheap high-memory validation workers in well under a second for typical podcast and video assets. The size-versus-bitrate guard short-circuits gross truncation before the packet walk, and PROBE_TIMEOUT bounds the worst case so one pathological file can't stall the pool.

What tolerance should I use for declared vs decoded duration?

Use the larger of an absolute floor (about 2 seconds) and a small fraction of the declared duration (around 2%). Container rounding and VBR estimation produce sub-second differences on perfectly valid files, so a tolerance tighter than that generates false positives; a real truncation is off by minutes, not seconds, so it clears the threshold easily.