Extracting Keyframe Timestamps from MP4

This page sits beneath Video Container Parsing with Python within the broader Media Ingestion & Format Architecture pipeline, and it isolates one primitive that everything downstream of ingestion quietly depends on: the exact wall-clock time of every keyframe in a video. A keyframe (an I-frame) is the only kind of frame a decoder can start from without prior context, so the set of keyframe timestamps is the set of clean cut points in the file. Get them and you can align packaging segments, build accurate seek indexes, and cut without re-encoding. Miss them — or read them in the wrong time base — and every segment boundary lands mid-GOP, forcing a decoder to guess and a packager to re-encode.

Problem Framing

The trap is that an MP4’s raw timestamps are not seconds. Each stream carries a time_base — a rational like 1/15360 — and a frame’s presentation timestamp (PTS) is an integer count of those ticks. Read the integer and treat it as seconds and your keyframe at 6.0 seconds reports as 92,160. Worse, video frames are stored in decode order, not presentation order, because B-frames reference frames that come after them on screen; sort by the wrong field and your “keyframe timeline” is scrambled. The symptoms are unmistakable once you know them:

INFO  keyframes  ep=ep0311  first_kf_pts=0 second_kf_pts=92160 reason="raw PTS ticks, not seconds"
WARN  keyframes  ep=ep0311  gop=irregular deltas=[2.0,2.0,6.0,2.0] reason="scene-cut I-frames; not fixed GOP"
WARN  segment    ep=ep0311  boundary=4.0s nearest_kf=6.0s reason="segment split mid-GOP -> re-encode forced"

The first line is the time-base miss: 92160 / 15360 = 6.0, so the keyframe is at six seconds, but only after you divide by the time base. The second is a real-world subtlety: many encoders insert extra keyframes at scene changes on top of the fixed cadence, so the interval between keyframes is not constant — you must read the actual list, never assume “one every N seconds.” The third is the payoff for getting it right or the penalty for getting it wrong: a packager that splits a segment at 4.0s when the nearest keyframe is at 6.0s cannot produce an independently-decodable segment, so it either re-encodes (expensive) or produces a segment that stutters on seek. This page shows both extraction paths — the fast ffprobe scan and the in-process PyAV decode — and the exact PTS-to-seconds conversion that makes the numbers usable.

Solution Architecture

There are two ways to enumerate keyframes and they trade speed for control. The ffprobe path uses -skip_frame nokey, which tells the decoder to skip every non-keyframe, so it walks the file cheaply and prints only I-frame packets with their timestamps — ideal for a fast index. The PyAV path opens the container in-process, iterates decoded frames, and filters on the key_frame flag, giving you the frame object itself (useful when you also want to thumbnail the keyframe or inspect picture type). Both funnel into the same normalization: multiply each keyframe’s pts by its stream time_base to get seconds, sort by presentation time, and emit a monotonic list of keyframe seconds plus the GOP deltas between them.

The diagram makes the load-bearing fact concrete: keyframes are not evenly spaced (GOP 1 is six seconds, GOP 2 only two), so any consumer that hard-codes an interval will drift. Read the real list and both consumers — the seek index and the segment aligner — operate on ground truth.

Implementation

1. Enumerate keyframes with ffprobe

The fast path uses -skip_frame nokey so the decoder discards every non-keyframe and reports only I-frames. Ask for pkt_pts_time (already in seconds) and pict_type, and select just the video stream. Parsing the JSON is trivial and this scan is I/O-bound, not decode-bound, because the non-keyframes are never reconstructed:

# Python 3.10+
import json
import subprocess

def keyframes_ffprobe(path: str) -> list[float]:
    """Return sorted keyframe presentation times in seconds via ffprobe."""
    proc = subprocess.run(
        ["ffprobe", "-hide_banner", "-v", "error",
         "-skip_frame", "nokey",          # decode only keyframes
         "-select_streams", "v:0",
         "-show_entries", "frame=pts_time,pict_type",
         "-of", "json", path],
        capture_output=True, text=True, check=True,
    )
    frames = json.loads(proc.stdout).get("frames", [])
    times = [
        float(f["pts_time"])
        for f in frames
        if f.get("pict_type") == "I" and f.get("pts_time") not in (None, "N/A")
    ]
    return sorted(times)

pts_time is the presentation timestamp already converted to seconds by ffprobe, so this path sidesteps the time-base arithmetic entirely — which is exactly why it is the right default for a bulk index. Filtering on pict_type == "I" is belt-and-suspenders; with -skip_frame nokey the decoder should only surface keyframes, but an occasional container reports a non-key packet and the explicit filter drops it.

2. Decode keyframes in-process with PyAV

When you need the frame object itself — to grab a thumbnail at each cut, or to distinguish an IDR frame from a plain I-frame — decode in-process with PyAV and read frame.pts directly. Here you do own the time-base conversion, and PyAV exposes it as a Fraction on the stream:

# av==12.0.0
import av

def keyframes_pyav(path: str) -> list[float]:
    """Return sorted keyframe times in seconds using PyAV frame decoding."""
    times: list[float] = []
    with av.open(path, mode="r") as container:
        stream = container.streams.video[0]
        # Ask the demuxer to hand us only keyframe packets where possible.
        stream.codec_context.skip_frame = "NONKEY"
        time_base = float(stream.time_base)   # Fraction -> float, e.g. 1/15360
        for frame in container.decode(stream):
            if frame.key_frame and frame.pts is not None:
                times.append(frame.pts * time_base)   # ticks -> seconds
    return sorted(times)

Setting skip_frame = "NONKEY" on the codec context is the PyAV equivalent of -skip_frame nokey: it lets libav drop non-keyframe reconstruction so the loop is cheap. The multiplication frame.pts * time_base is the whole conversion — 92160 * (1/15360) = 6.0 — and doing it with the stream’s real time_base rather than an assumed 1/1000 or 1/90000 is what keeps the seconds correct across differently-authored files. The tradeoff between this in-process path and the subprocess scan is quantified in FFmpeg vs PyAV for video ingestion.

3. Derive GOP deltas and expose the contract

Both paths return the same sorted list of seconds; wrap it in a small contract that also reports the inter-keyframe deltas, because a packager cares as much about GOP regularity as about the timestamps themselves:

# Python 3.10+
def keyframe_report(times: list[float]) -> dict:
    deltas = [round(b - a, 3) for a, b in zip(times, times[1:])]
    return {
        "keyframe_count": len(times),
        "keyframes_s": [round(t, 3) for t in times],
        "gop_deltas_s": deltas,
        "regular_gop": len(set(deltas)) <= 1,   # True only if perfectly fixed
        "max_gop_s": max(deltas) if deltas else 0.0,
    }

max_gop_s is the number the packaging layer actually consumes: a segment duration must be an integer multiple of the GOP length, and if scene-cut keyframes make the GOP irregular, the maximum gap is the binding constraint. This is the direct input to segment alignment in HLS packaging — the adaptive bitrate packaging layer snaps every segment boundary to a keyframe time, and tuning HLS segment duration for seek latency uses exactly this max_gop_s to pick a segment length that never splits a GOP. The -skip_frame semantics are documented in the official FFmpeg documentation.

Verification

Confirm the two extraction paths agree and that the numbers are real seconds, not ticks.

# 1. The ffprobe and PyAV paths must return the same keyframe list.
python -c "from keyframes import keyframes_ffprobe, keyframes_pyav as p; \
a=keyframes_ffprobe('clip.mp4'); b=p('clip.mp4'); \
print(len(a), len(b), all(abs(x-y) < 0.01 for x,y in zip(a,b)))"
# expected: matching counts and True

# 2. Ground-truth the first few keyframes directly from ffprobe packet flags.
ffprobe -hide_banner -v error -select_streams v:0 -skip_frame nokey \
  -show_entries frame=pts_time,pict_type -of csv=p=0 clip.mp4 | head

# 3. Sanity-check that timestamps are seconds, not ticks (first delta ~ GOP length).
python -c "from keyframes import keyframes_ffprobe, keyframe_report; \
print(keyframe_report(keyframes_ffprobe('clip.mp4')))"
# expected: gop_deltas_s like [2.0, 2.0, ...] or [6.0, ...], NOT [92160, ...]

# 4. Confirm the encoder's declared GOP matches what you extracted.
ffprobe -hide_banner -v error -select_streams v:0 \
  -show_entries stream=avg_frame_rate,time_base -of default=nw=1 clip.mp4

A correct run shows both extraction paths returning identical keyframe counts and times within a hundredth of a second, GOP deltas measured in single-digit seconds rather than tens of thousands, and a max_gop_s that a packager can divide segment durations by cleanly. Emit keyframe_count, max_gop_s, and regular_gop per asset to your observability stack so an upstream encoder change that lengthens the GOP — and would silently break segment alignment — is caught at ingest rather than at packaging.

Failure Modes & Edge Cases

Because keyframe extraction is a pure read over container structure, it slots into the stateless parse gate described in the parent video container parsing with Python guide, and any file whose keyframe scan fails is diverted by media validation and error routing before it reaches a packaging worker.

FAQ

Related

• Up to the parent guide: Video Container Parsing with Python
• Sibling on binding selection: FFmpeg vs PyAV for video ingestion
• Where these keyframes drive segment boundaries: adaptive bitrate packaging
• Picking a segment length from the GOP: tuning HLS segment duration for seek latency
• Pipeline overview: Media Ingestion & Format Architecture