Dockerizing Media Processing Containers

A GPU pool additionally requires the NVIDIA Container Toolkit installed on the host and a driver new enough for the encoder session count you intend to run. CPU-only pools (diarization, metadata, audio normalization) set NVIDIA_VISIBLE_DEVICES=none and never load the CUDA libraries.

Architecture & Queue Topology

A media container is not a long-lived service; it is a short-lived worker that leases one job from the broker, validates it at the boundary, runs a bounded FFmpeg process against a tmpfs scratch area, writes a content-addressed result, and exits. The image is built once and bound to a specific queue by the scheduler — GPU images consume only from the hardware-encode queue, CPU images from the normalization and metadata queues — which is how Celery task routing for video jobs keeps a CPU-bound transcode from ever landing on a worker without NVENC.

The hard architectural rule is that nothing crosses the container boundary unvalidated in either direction: the entrypoint rejects a malformed manifest before spawning FFmpeg, and the result is only published after its checksum and output contract are verified. This mirrors the gate discipline of the upstream media ingestion and format architecture stage, pushed down to the worker so a single corrupt asset cannot poison the queue.

Step-by-Step Implementation

1. Pin the base image by digest

Floating latest or even bookworm-slim by tag lets the system libraries shift under you between builds. Resolve the digest once and commit it.

# Resolve with: docker buildx imagetools inspect debian:bookworm-slim
FROM debian:bookworm-slim@sha256:2bc5c236e9b262645a323e9088dfa3bb1ecb16cc75811daf40a23a824d665be9 AS base

Verify the digest is what you expect before building:

docker buildx imagetools inspect debian:bookworm-slim --format '{{.Manifest.Digest}}'

2. Compile codecs in a builder stage, ship only the runtime

Keep the toolchain out of the final image. Compile FFmpeg and its codec libraries in a builder, then copy only the resulting binaries and shared objects into a slim runtime. Detailed size-tuning of this stage lives in Optimizing Docker Images for FFmpeg Workloads.

FROM base AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential pkg-config yasm nasm \
        libx264-dev libfdk-aac-dev libsox-dev ca-certificates curl \
    && rm -rf /var/lib/apt/lists/*
# FFmpeg 6.1.1 — pinned tarball, configured for the codecs we actually use
RUN curl -fsSL https://ffmpeg.org/releases/ffmpeg-6.1.1.tar.xz | tar -xJ \
    && cd ffmpeg-6.1.1 \
    && ./configure --enable-gpl --enable-nonfree \
        --enable-libx264 --enable-libfdk-aac --disable-doc --disable-debug \
    && make -j"$(nproc)" && make install DESTDIR=/opt/ffmpeg

FROM base AS runtime
COPY --from=builder /opt/ffmpeg/usr/local/bin/ffmpeg /usr/local/bin/ffmpeg
COPY --from=builder /opt/ffmpeg/usr/local/bin/ffprobe /usr/local/bin/ffprobe
RUN apt-get update && apt-get install -y --no-install-recommends \
        python3.11 python3-pip libx264-164 libfdk-aac2 libsox3 \
    && rm -rf /var/lib/apt/lists/*

Verify the final image exposes exactly the encoders you pinned:

docker run --rm media-worker:6.1.1 ffmpeg -hide_banner -encoders | grep -E 'libx264|libfdk_aac'

3. Pin the Python layer and drop privileges

Install Python dependencies from a locked manifest with hashes, then run as a non-root user with a read-only root filesystem so a compromised codec cannot mutate the image.

COPY requirements.lock /tmp/requirements.lock
RUN python3 -m pip install --no-cache-dir --require-hashes -r /tmp/requirements.lock
RUN useradd --uid 10001 --no-create-home --shell /usr/sbin/nologin worker
USER 10001:10001
ENTRYPOINT ["python3", "-m", "media_worker"]

Verify the container runs unprivileged:

docker run --rm media-worker:6.1.1 id -u   # -> 10001

4. Apply cgroups v2 resource limits at run time

Media jobs are resource-intensive; an unbounded container will destabilize a shared host. Launch every worker with explicit CPU and memory caps, disable swap (which injects unacceptable latency into real-time audio), and mount scratch as tmpfs to keep mux/demux buffers off disk.

docker run --rm \
  --cpus=4 --memory=4g --memory-swap=4g \
  --read-only \
  --tmpfs /scratch:rw,size=2g,mode=1777 \
  -e MEDIA_SCRATCH=/scratch -e FFMPEG_THREADS=4 -e CONTAINER_MEM_MB=4096 \
  media-worker:6.1.1

Setting --memory-swap equal to --memory disables swap usage entirely. Verify the kernel actually applied the ceiling from inside the container:

docker exec <id> cat /sys/fs/cgroup/memory.max   # -> 4294967296

5. Expose the GPU to encode pools only

GPU workers add the NVIDIA runtime and expose a single device, never full passthrough. The NVIDIA Container Toolkit wires the driver and CUDA libraries into the container at launch.

docker run --rm --gpus '"device=0"' \
  --cpus=6 --memory=8g --memory-swap=8g \
  -e NVIDIA_VISIBLE_DEVICES=0 \
  media-worker-gpu:6.1.1 ffmpeg -hwaccels

Verify the GPU is visible and NVENC is available:

docker run --rm --gpus '"device=0"' media-worker-gpu:6.1.1 nvidia-smi --query-gpu=name --format=csv,noheader

6. Add a codec-aware HEALTHCHECK

A TCP port check tells you nothing about a media worker. Probe codec initialization and library loading instead, so the orchestrator only schedules onto images whose encoder actually loads.

HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD ffmpeg -hide_banner -loglevel error -f lavfi -i anullsrc \
      -c:a aac -t 0.1 -f null - || exit 1

Verify the orchestrator sees the container as healthy:

docker inspect --format '{{.State.Health.Status}}' <id>   # -> healthy

7. Validate the job manifest at the entrypoint

The worker rejects a malformed manifest before spawning FFmpeg, returning a deterministic exit code the scheduler can route on. This is the contract boundary the rest of the pipeline relies on.

# media_worker/__main__.py
# pydantic==2.6.4, ffmpeg-python==0.2.0
import hashlib
import os
import subprocess
import sys
from pydantic import BaseModel, Field, ValidationError

class JobManifest(BaseModel):
    schema_version: str = Field(pattern=r"^v3$")
    sha256: str = Field(min_length=64, max_length=64)
    input_uri: str
    target_codec: str  # one of: libx264, libfdk_aac
    target_bitrate: str  # e.g. "192k"

def load_manifest(raw: str) -> JobManifest:
    try:
        return JobManifest.model_validate_json(raw)
    except ValidationError as exc:
        sys.stderr.write(exc.json())
        sys.exit(65)  # EX_DATAERR: permanent, do not retry

def probe(path: str) -> None:
    # ffprobe gate: a non-decodable input is a permanent failure
    rc = subprocess.run(
        ["ffprobe", "-v", "error", "-show_streams", path],
        capture_output=True,
    ).returncode
    if rc != 0:
        sys.exit(65)

Verify the gate rejects bad input with the permanent-failure code:

echo '{"schema_version":"v3"}' | docker run --rm -i media-worker:6.1.1
echo $?   # -> 65

Data Contracts

The container consumes a job manifest and emits a result manifest. Both are schema-validated at the boundary; a payload missing any required field is rejected before compute begins.

Recording the image_digest on every result is what makes the reproducibility SLA auditable: a re-run on the same digest with the same sha256 input must yield the same output_sha256.

Resilience Patterns

The container itself stays stateless and idempotent; durability is a contract between its exit codes and the broker. Outputs are written to a content-addressed path keyed on the input sha256, so a redelivered message that finds a completed artifact re-acknowledges its lease instead of re-encoding — at-least-once delivery can never double-publish an episode.

Exit codes are the routing signal. The worker distinguishes transient failures (exit 75, EX_TEMPFAIL — storage timeout, transient codec lock) from permanent ones (exit 65, EX_DATAERR — malformed manifest, undecodable input, codec mismatch). The scheduler retries transient codes with exponential backoff and jitter, and routes permanent codes straight to a dead-letter queue with the original asset and failure metadata preserved. The full backoff and quarantine machinery is owned by the retry logic and dead letter queues stage; the container’s only job is to fail closed with the correct, deterministic exit code so that stage can act on it. When the worker is driven from a DAG, those same codes gate progression in Orchestrating Pipelines with Airflow, where a non-zero exit fails the task rather than silently advancing.

Observability & Debugging

A containerized worker is only as trustworthy as its telemetry. Emit structured JSON to stdout/stderr so it flows into Docker’s logging driver and the central aggregator without parsing heuristics, and expose Prometheus metrics for queue-relevant signals.

# prometheus-client==0.20.0
from prometheus_client import Counter, Gauge, start_http_server

TRANSCODE_SECONDS = Gauge("media_transcode_seconds", "Wall-clock per job", ["codec"])
FRAME_DROPS = Counter("media_frame_drops_total", "Decoder frame drops", ["codec"])
RESTARTS = Counter("media_container_restarts_total", "OOM/health restarts")

start_http_server(9100)  # scraped per worker

Every log line carries job_id, sha256, image_digest, queue_name, and exit_code so a failure can be correlated across the broker, the orchestrator, and storage. For live debugging, prefer an ephemeral sidecar sharing the worker’s PID namespace (--pid=container:<id>) over docker exec into a read-only production container.

Performance Tuning

Throughput on media containers is governed by buffer sizing and thread discipline, not raw CPU count. Size the memory ceiling against peak frame-buffer demand plus 15–20% headroom for codec lookahead buffers and Python garbage-collection pauses — a 1080p H.264 encode is comfortable at --memory=4g, while 4K HDR typically needs --memory=8g to avoid OOM kills. Cap FFmpeg’s threads explicitly (-threads/FFMPEG_THREADS) and set OMP_NUM_THREADS to match, because letting both libraries spawn one thread per core on a high-density host oversubscribes the CPU and raises per-job latency without lifting aggregate throughput.

Mount scratch as tmpfs sized to the largest intermediate the job class produces — undersized scratch forces FFmpeg to spill to the read-only layer and stalls. On GPU pools, the binding constraint is the encoder session limit of the card, not VRAM: schedule concurrency to the NVENC session count rather than to core count, and keep CPU-bound demux work on separate CPU pools via Celery task routing for video jobs so a transcode never starves a parallel audio-normalization job.

Frequently Asked Questions

Related

• Up to the parent overview: Pipeline Automation & Batch Processing
• Shrinking and tuning the image: Optimizing Docker Images for FFmpeg Workloads
• Binding these images to worker pools: Celery task routing for video jobs
• Scheduling containers across a DAG: Orchestrating Pipelines with Airflow
• Where non-zero exits are absorbed: retry logic and dead letter queues