Parallel FFmpeg Encoding with a Worker Pool

This page sits beneath FFmpeg Batch Processing for Podcasts within the broader Media Ingestion & Format Architecture pipeline, and it answers one deceptively simple question: when a batch of episodes lands and you want to encode them all at once, how many ffmpeg processes should actually run — and how do you stop them from fighting each other for the same cores? Naive parallelism is the classic own-goal here. Launching one process per file feels fast, but on a machine with sixteen cores and forty queued episodes, forty FFmpeg processes each spinning up its own decode/filter/encode threads produces a load average north of 200 and finishes slower than a bounded pool of eight.

Problem Framing

The failure is not a crash — it is a slow, invisible collapse of throughput that the process exit codes never report. Every FFmpeg invocation defaults -threads 0, meaning “use as many threads as there are logical CPUs.” So each process believes it owns the whole machine. Stack a dozen of them and the kernel scheduler spends its time context-switching between hundreds of runnable threads instead of doing FPU work. The tell is a realtime factor that climbs under load while CPU utilization sits pinned at 100% — you are busy, but you are busy thrashing:

INFO  encode  ep=ep0207  workers=40  load1=214.6  rtf=1.88   # 40 unbounded procs
INFO  encode  ep=ep0207  workers=8   load1=15.2   rtf=0.41   # bounded pool, -threads 2
WARN  scheduler  ctx_switches=1.4M/s  reason="thread oversubscription; runnable>>cores"

The second line is the target: eight worker processes, each pinned to two threads, on a sixteen-core host, finish the same cohort in less than a quarter of the wall-clock time. The arithmetic that governs this is pool_size × threads_per_ffmpeg ≤ physical_cores. Break that inequality and you oversubscribe; obey it and every process gets a clean slice. On top of core budgeting you need two safety rails the unbounded approach lacks: a per-job timeout so a single pathological file cannot pin a worker forever, and a cgroup/nice boundary so the encode pool cannot starve the dispatcher, the broker client, or ffprobe running on the same box. This page specifies all three.

Solution Architecture

The design is a single bounded ProcessPoolExecutor sitting behind a job queue. The pool size is fixed at startup from the core budget, not from the number of pending jobs — so a spike in submissions grows the queue, never the process count. Each worker runs exactly one ffmpeg at a time with an explicit -threads value chosen so the product stays inside the physical core count. A watchdog wraps every subprocess with a hard timeout, killing the process group on breach so no half-encoded output escapes. And the whole worker is launched under a cgroup CPU quota with a raised nice value, so even under a scheduling anomaly the encode pool yields to the coordinator process that feeds it.

Bounded worker-pool dispatch for parallel FFmpeg encoding on a 16-core host A left-to-right diagram. A job queue of pending episodes feeds a bounded dispatcher that holds a semaphore of size four. The semaphore admits at most four concurrent workers into a ProcessPoolExecutor; excess jobs wait in the queue. Each of the four worker slots runs one ffmpeg process pinned to four threads, so four workers times four threads equals the sixteen physical cores exactly, with no oversubscription. Each worker is wrapped by a timeout watchdog that kills the process group on breach and is launched under a cgroup CPU quota with a raised nice value. Completed encodes exit to a result sink; timed-out jobs route to a dead-letter path. Job queue 40 episodes pending encode Dispatcher semaphore = 4 admits ≤ 4 workers ProcessPoolExecutor 4 WORKERS × 4 THREADS = 16 CORES worker 1 ffmpeg -threads 4 worker 2 ffmpeg -threads 4 worker 3 ffmpeg -threads 4 worker 4 ffmpeg -threads 4 Per-worker guardrails timeout watchdog · kill process group on breach cgroup CPU quota · nice +5 Result sink encoded m4a Dead-letter timed-out job done timeout

The key insight the diagram encodes is that concurrency lives in two places — the Python pool and FFmpeg’s internal thread count — and they multiply. Tuning one while ignoring the other is why a “properly bounded” pool of sixteen workers still melts a sixteen-core box: each worker was quietly spawning sixteen threads of its own.

Implementation

1. Budget cores across pool size and threads

Start from the physical core count and pick a split. For CPU-bound audio and video encodes the sweet spot is usually a wider pool with fewer threads per process, because FFmpeg’s threading scales sub-linearly past three or four threads on a single encode. A common, robust split on a sixteen-core host is four workers of four threads. Compute both dials from one source of truth so they can never drift out of the inequality:

# Python 3.10+
# psutil==5.9.8   (physical-core count; falls back to os.cpu_count())
import os
import psutil
from dataclasses import dataclass

@dataclass(frozen=True)
class PoolBudget:
    pool_size: int
    threads_per_job: int

    def __post_init__(self) -> None:
        cores = psutil.cpu_count(logical=False) or os.cpu_count() or 1
        if self.pool_size * self.threads_per_job > cores:
            raise ValueError(
                f"oversubscribed: {self.pool_size}x{self.threads_per_job} "
                f"= {self.pool_size * self.threads_per_job} > {cores} physical cores"
            )

def plan_budget(threads_per_job: int = 4) -> PoolBudget:
    cores = psutil.cpu_count(logical=False) or os.cpu_count() or 1
    pool_size = max(1, cores // threads_per_job)
    return PoolBudget(pool_size=pool_size, threads_per_job=threads_per_job)

The __post_init__ guard is deliberately fatal: a misconfiguration that would oversubscribe the host should stop the worker at startup, not degrade silently in production. psutil.cpu_count(logical=False) returns physical cores, which is what you want — hyperthreads do not help an FPU-saturated loudnorm/soxr chain and counting them is the single most common cause of the thrash shown above.

2. Dispatch through a bounded pool with per-job timeouts

ProcessPoolExecutor gives you the bounded pool for free; the missing piece is a hard wall-clock ceiling per job. Pass FFmpeg the chosen -threads explicitly and wrap the subprocess in subprocess.run(..., timeout=...), then kill the whole process group on breach so a hung encode leaves no orphaned children:

# Python 3.10+
import subprocess
from concurrent.futures import ProcessPoolExecutor, as_completed

def encode_one(job: dict, threads: int, timeout_s: int) -> dict:
    cmd = [
        "ffmpeg", "-hide_banner", "-nostdin", "-y",
        "-threads", str(threads),
        "-i", job["input"],
        "-c:a", "aac", "-b:a", "192k", "-ar", "48000",
        job["output"],
    ]
    try:
        proc = subprocess.run(
            cmd, capture_output=True, text=True,
            timeout=timeout_s, check=True,
            start_new_session=True,   # own process group -> killable as a unit
        )
        return {"sha256": job["sha256"], "status": "success"}
    except subprocess.TimeoutExpired:
        return {"sha256": job["sha256"], "status": "timeout"}
    except subprocess.CalledProcessError as exc:
        return {"sha256": job["sha256"], "status": "failed",
                "stderr": exc.stderr[-2000:]}

def run_pool(jobs: list[dict], budget: PoolBudget, per_job_timeout_s: int = 1800) -> list[dict]:
    results = []
    with ProcessPoolExecutor(max_workers=budget.pool_size) as pool:
        futures = {
            pool.submit(encode_one, j, budget.threads_per_job, per_job_timeout_s): j
            for j in jobs
        }
        for fut in as_completed(futures):
            results.append(fut.result())
    return results

-nostdin prevents a worker FFmpeg from blocking on a stdin read when it is not attached to a terminal — a subtle hang that looks exactly like a slow encode. Set per_job_timeout_s from the longest legitimate asset times a safety multiple: for six-hour long-form audio at a realtime factor of 0.4, roughly 1800 seconds is a generous ceiling. Jobs that breach it are returned as timeout and handed to the retry logic and dead-letter queue machinery rather than retried in place.

3. Constrain each worker with nice and a cgroup limit

The pool math protects the host from the encode pool oversubscribing itself, but it does not stop the pool from starving the coordinator process, the Redis client, or an ffprobe pre-flight sharing the box. Two cheap boundaries fix that. Launch each FFmpeg with a raised nice value so the scheduler always favours the lightweight coordinator, and wrap the worker in a cgroup (v2) CPU quota so its aggregate usage is hard-capped even under a runaway filter graph:

# Prefix the ffmpeg command with nice; the coordinator stays at nice 0.
cmd = ["nice", "-n", "5", *cmd]
# cgroup v2: cap the encode pool at 15 of 16 cores, leaving one for the coordinator.
sudo mkdir -p /sys/fs/cgroup/ffmpeg-pool
echo "1500000 100000" | sudo tee /sys/fs/cgroup/ffmpeg-pool/cpu.max   # 15 cores
# launch the worker process inside the cgroup
echo $$ | sudo tee /sys/fs/cgroup/ffmpeg-pool/cgroup.procs

The cpu.max value 1500000 100000 means “1,500,000 microseconds of CPU per 100,000-microsecond period” — fifteen cores’ worth — which reserves one physical core for everything that is not encoding. That reservation is what keeps queue leasing and metric emission responsive while the pool runs hot. When you package this worker as a container, the same cap is expressed with --cpus=15 and the image-size choices that keep cold starts fast are covered in optimizing Docker images for FFmpeg workloads. Detailed FFmpeg threading semantics live in the official FFmpeg documentation.

Verification

Prove the pool is bounded and that the product stays inside the core count under real load — a pool that is bounded on paper can still oversubscribe if -threads was left at its default.

# 1. Assert the budget guard rejects an oversubscribing plan.
python -c "from pool import PoolBudget; PoolBudget(pool_size=16, threads_per_job=4)"
# expected: ValueError 'oversubscribed: 16x4 = 64 > 16 physical cores'

# 2. Run a full cohort and watch load average track core count, not job count.
python run_batch.py --jobs 40 &
uptime            # load1 should settle near 15, NOT near 40
nproc             # confirm the physical/logical core count you budgeted against

# 3. Count live ffmpeg processes at saturation — must equal pool_size, never job count.
watch -n1 'pgrep -c -x ffmpeg'   # steady-state = 4, regardless of 40 queued

# 4. Confirm the timeout actually kills the process group (no orphaned children).
python -c "from pool import run_pool, plan_budget; \
print(run_pool([{'input':'hang.wav','output':'/tmp/o.m4a','sha256':'x'}], plan_budget(), 5))"
# expected: [{'sha256':'x','status':'timeout'}] and 'pgrep -x ffmpeg' returns nothing after

A healthy run shows exactly pool_size live FFmpeg processes at any instant, a one-minute load average that hovers around the core count rather than the backlog depth, and a realtime factor that stays flat as the queue drains. Export pool_active_workers, encode_realtime_factor, and job_timeout_total to your observability stack so a creeping realtime factor — the early warning of oversubscription after a host change — is caught before it doubles your batch window.

Failure Modes & Edge Cases

Edge case Symptom Remediation
-threads left at default (0) Each worker spawns cores-worth of threads; load average explodes Pass -threads explicitly; assert pool_size × threads ≤ cores at startup
Pool sized to logical (hyperthread) count Realtime factor climbs under load, throughput flat Size against psutil.cpu_count(logical=False); hyperthreads do not help FPU work
No per-job timeout One corrupt file pins a worker indefinitely, effective pool shrinks Wrap every subprocess in subprocess.run(timeout=...) with start_new_session=True
Timeout kills process but not children Orphaned ffmpeg child keeps burning CPU after the parent dies Launch with start_new_session=True and kill the whole process group
Encode pool starves the coordinator Queue leasing and metrics stall while jobs run Reserve one core via cpu.max and raise encode nice to +5
Pool size larger than pending jobs Idle workers hold memory but do no work Cap max_workers at min(pool_size, len(jobs)) for small batches
Very short clips dominate the batch Process fork/teardown overhead exceeds encode time Batch tiny assets into one FFmpeg invocation with a concat list

For long-running batches, decouple the pool from any synchronous request cycle and lease jobs from a broker so a spike in submissions grows queue depth instead of the process count — the queue topology this pool plugs into is specified in the parent FFmpeg batch processing for podcasts guide.

FAQ

Should I use more workers with fewer -threads, or fewer workers with more threads?

Prefer more workers with fewer threads for a batch of independent files. FFmpeg's internal threading scales sub-linearly — a single encode rarely gets much past 3–4x from added threads — whereas running independent files in parallel scales almost linearly until you hit the core count. On a 16-core host, four workers of four threads beats two workers of eight for aggregate throughput on a deep queue. Fall back to wide-threads-per-job only when the queue is shallow and you want each individual file to finish sooner.

Why ProcessPoolExecutor and not ThreadPoolExecutor or asyncio?

Each FFmpeg encode is an external subprocess, so the Python-level concurrency primitive only needs to launch and await processes — it never touches the CPython GIL for the actual work. A thread pool or an asyncio task group would work equally well for that, but a process pool gives you clean per-worker isolation and a natural place to hang the cgroup and nice boundaries. The heavy lifting happens in the child ffmpeg processes regardless; the pool is just a bounded dispatcher.

How do I pick the per-job timeout without killing legitimate long files?

Base it on your longest legitimate asset multiplied by a realistic worst-case realtime factor, then add headroom. If your longest episode is six hours and a saturated worker runs at a realtime factor around 0.4, a single file needs roughly 8,600 seconds of wall clock in the worst case — so a ceiling near that plus a safety margin avoids false kills. Track the actual encode-duration histogram and set the timeout above the p99.9, not the mean, so only genuinely stuck jobs are terminated.