Setting Up Fallback Routing for Failed Transcodes

Within the Media Validation & Error Routing layer of the broader Media Ingestion & Format Architecture pipeline, this page solves one exact production scenario: a GPU-accelerated transcode dies mid-batch — a driver timeout, an unsupported pixel format, a VRAM exhaustion — and you need the job to degrade to a software encoder automatically instead of stalling the queue or paging an engineer at 3am.

Problem Framing

Automated podcast and video pipelines lean on hardware encoders like NVIDIA NVENC (h264_nvenc, hevc_nvenc) or AMD AMF because they are 5–10x faster than software encoding. The cost of that speed is a brittle failure surface: a single h264_nvenc worker that loses its CUDA context will fail every job routed to it until the driver is reset, and a naïve batch loop will happily feed the whole queue into that black hole. The symptom in your worker log looks like this:

[h264_nvenc @ 0x55e3c0] Cannot load libcuda.so.1
[h264_nvenc @ 0x55e3c0] Failed loading nvcuda.
[h264_nvenc @ 0x55e3c0] The minimum required Nvidia driver for nvenc is 520.56.06 or newer
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0
Conversion failed!

FFmpeg exits non-zero, the job is marked failed, and the next job inherits the same dead context. On a dashboard you see it as a sawtooth: transcode_failures_total spikes, queue depth climbs, and GPU utilization drops to zero while CPU sits idle. The fix is not “retry on the same encoder” — that just burns the max_retry_attempts budget against a deterministic hardware fault. The fix is a router that classifies why the job failed and, for compute-bound faults, re-runs it on libx264/libfdk_aac before the failure cascades. This is the same deterministic-degradation discipline the retry logic and dead-letter queues layer applies one tier up — here it is specialized to the encoder boundary.

Solution Architecture

The router wraps each FFmpeg invocation in an asyncio subprocess manager. On a non-zero exit it reads the last N lines of stderr, matches them against a table of known failure signatures, and decides between three outcomes: success (emit), recoverable hardware fault (swap encoder, re-run once), or fatal input fault (abort and quarantine — no fallback can fix a missing moov atom). Container-level corruption is intentionally not something the router retries; that belongs upstream in the validation gate documented in handling corrupt MP4 files in automated pipelines, so the fallback router can focus exclusively on compute-bound failures.

Fallback router state machine: classify a transcode failure, then emit, swap to software, or quarantine A vertical state diagram. The primary GPU attempt on h264_nvenc feeds a classify step that reads the exit code and the bounded stderr tail. A clean exit routes to Emit. A fatal container fault routes left to Quarantine with no fallback. A recoverable compute fault — CUDA_INIT, OOM_KILLED, or PROFILE_MISMATCH — flows down through a fallback-delay sleep that lets the GPU context tear down, then swaps to the libx264 software encoder for a single re-run. That re-run is classified again: success routes to Emit, while a second failure aborts the job. The fallback path is capped at one attempt by max_retries equals one. Primary attempt h264_nvenc · GPU fast path classify() exit_code + stderr tail last 50 lines Emit durable write, then ack Quarantine CONTAINER_DEADLOCK sleep(fallback_delay) 2s · GPU context tear-down Swap encoder, re-run libx264 · software safety net classify() re-run result Emit software-encoded H.264 Abort → DLQ fallback exhausted exit 0 · no signature fatal recoverable 2nd failure ok CUDA_INIT · OOM_KILLED PROFILE_MISMATCH max_retries = 1 one fallback run

The failure taxonomy below drives every routing decision. Tune the thresholds at the orchestration layer before invoking the router:

Failure mode Signature (exit + stderr tail) Root cause Routing decision
CUDA_INIT Failed to initialize CUDA, Cannot load libcuda.so, nvenc … init … error Driver mismatch, stale GPU context, insufficient VRAM Swap to software encoder, re-run once
PROFILE_MISMATCH Error while opening encoder, Profile/level not supported 10-bit HDR source into an 8-bit constrained encoder Swap to software encoder, re-run once
OOM_KILLED signal 9, Out of memory, OOMKilled VRAM > ~85% during multi-pass encode Swap to software encoder, re-run once
CONTAINER_DEADLOCK Invalid data found when processing input, moov atom not found Corrupt/truncated container Abort → quarantine (no fallback)
TIMEOUT_EXCEEDED wall-clock > timeout Hung process, livelock Kill, abort, alert

Baseline thresholds that keep this deterministic: transcode_timeout of 300s (or 3.0x the expected duration derived from input bitrate), max_retry_attempts of 1 to prevent cascading failures on corrupted inputs, a stderr_match_window of the last 50 lines to suppress false positives from benign FFmpeg warnings, and a fallback_delay of 2s to let the GPU driver context tear down before the software job starts.

Implementation

The router below is complete and copy-pasteable. It uses asyncio.create_subprocess_exec so a worker can supervise many transcodes concurrently, enforces a hard timeout, classifies failures against the table above, and tracks state so the same job is never retried on a dead encoder. Every tunable lives on the TranscodeConfig dataclass.

# Python 3.10+
# (stdlib only: asyncio, logging, re, dataclasses — no third-party deps)
# Requires ffmpeg >= 5.1 on PATH, built with --enable-nvenc and --enable-libx264
import asyncio
import logging
import re
from dataclasses import dataclass, field
from typing import Optional, Tuple

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(name)s: %(message)s"
)
logger = logging.getLogger("fallback_router")

@dataclass
class TranscodeConfig:
    input_path: str
    output_path: str
    primary_encoder: str = "h264_nvenc"   # hardware-accelerated, fast path
    fallback_encoder: str = "libx264"     # software, always-available safety net
    timeout: float = 300.0                # hard wall-clock ceiling, seconds
    max_retries: int = 1                  # 1 == primary attempt + one fallback attempt
    stderr_window: int = 50               # only classify against the last N stderr lines
    fallback_delay: float = 2.0           # let the GPU driver context tear down first

class FallbackRouter:
    def __init__(self, config: TranscodeConfig):
        self.config = config
        self.state = {
            "attempts": 0,
            "fallback_triggered": False,
            "failure_mode": None,
            "diagnostics": []
        }

    def _build_ffmpeg_cmd(self, encoder: str) -> list[str]:
        # -y overwrites; -preset medium is a balanced CPU/quality tradeoff on libx264.
        # For audio-only pipelines, swap the video flags for `-af loudnorm` + an
        # audio encoder (e.g. libfdk_aac) — the fallback logic is identical.
        return [
            "ffmpeg", "-y", "-i", self.config.input_path,
            "-c:v", encoder, "-preset", "medium",
            "-c:a", "aac", "-b:a", "192k",
            self.config.output_path
        ]

    async def _run_ffmpeg(self, cmd: list[str]) -> Tuple[int, str, str]:
        try:
            proc = await asyncio.create_subprocess_exec(
                *cmd,
                stdout=asyncio.subprocess.PIPE,
                stderr=asyncio.subprocess.PIPE
            )
            stdout, stderr = await asyncio.wait_for(
                proc.communicate(), timeout=self.config.timeout
            )
            return (
                proc.returncode or 0,
                stdout.decode("utf-8", errors="replace"),
                stderr.decode("utf-8", errors="replace"),
            )
        except asyncio.TimeoutError:
            try:
                proc.kill()        # never leave a hung encoder holding the GPU
            except ProcessLookupError:
                pass
            logger.error("Transcode timed out after %.1fs", self.config.timeout)
            return -1, "", "TIMEOUT_EXCEEDED"
        except Exception as e:
            logger.error("Subprocess execution failed: %s", e)
            return -2, "", f"SUBPROCESS_ERROR: {e}"

    def _classify_failure(self, exit_code: int, stderr: str) -> Optional[str]:
        # Match only against the bounded tail so a benign warning 4000 lines up
        # cannot trigger a spurious fallback.
        tail = "\n".join(stderr.strip().splitlines()[-self.config.stderr_window:])
        patterns = {
            "CUDA_INIT": re.compile(
                r"(Failed to initialize CUDA|Cannot load libcuda\.so|nvenc.*init.*error)", re.I),
            "PROFILE_MISMATCH": re.compile(
                r"(Error while opening encoder|Profile/level not supported)", re.I),
            "CONTAINER_DEADLOCK": re.compile(
                r"(Invalid data found when processing input|moov atom not found)", re.I),
            "OOM_KILLED": re.compile(
                r"(Out of memory|signal 9|OOMKilled)", re.I),
        }
        for mode, pattern in patterns.items():
            if pattern.search(tail):
                return mode
        return "GENERIC_FAILURE" if exit_code != 0 else None

    async def execute(self) -> bool:
        for attempt in range(self.config.max_retries + 1):
            self.state["attempts"] = attempt + 1
            encoder = (
                self.config.fallback_encoder
                if self.state["fallback_triggered"]
                else self.config.primary_encoder
            )
            cmd = self._build_ffmpeg_cmd(encoder)

            logger.info("Attempt %d/%d | Encoder: %s",
                        self.state["attempts"], self.config.max_retries + 1, encoder)
            exit_code, stdout, stderr = await self._run_ffmpeg(cmd)
            failure_mode = self._classify_failure(exit_code, stderr)

            self.state["diagnostics"].append({
                "attempt": self.state["attempts"],
                "encoder": encoder,
                "exit_code": exit_code,
                "failure_mode": failure_mode,
                "stderr_tail": stderr[-200:] if stderr else None,
            })

            if failure_mode is None:
                self.state["failure_mode"] = None
                logger.info("Transcode succeeded on attempt %d.", self.state["attempts"])
                return True

            # Fatal input faults are never recoverable by swapping encoders.
            if failure_mode == "CONTAINER_DEADLOCK":
                self.state["failure_mode"] = failure_mode
                logger.error("Container fault for %s; routing to quarantine, no fallback.",
                             self.config.input_path)
                return False

            self.state["failure_mode"] = failure_mode
            logger.warning("Failure detected: %s (exit %d). Preparing fallback routing.",
                           failure_mode, exit_code)

            if not self.state["fallback_triggered"]:
                self.state["fallback_triggered"] = True
                logger.info("Routing to software fallback encoder: %s",
                            self.config.fallback_encoder)
                await asyncio.sleep(self.config.fallback_delay)
            else:
                logger.error("Fallback also failed. Aborting pipeline for %s.",
                             self.config.input_path)
                return False

        return False

    def get_diagnostics_report(self) -> dict:
        return {
            "input": self.config.input_path,
            "final_status": (
                "SUCCESS"
                if self.state["attempts"] > 0 and self.state["failure_mode"] is None
                else "FAILED"
            ),
            "failure_classification": self.state["failure_mode"],
            "execution_trace": self.state["diagnostics"],
        }

Wire it into a batch worker the same way you would any FFmpeg job — gate the input through validation first, then hand off to the router and forward the diagnostics report to your monitoring sink:

async def transcode_one(media_id: str, src: str, dst: str) -> dict:
    cfg = TranscodeConfig(input_path=src, output_path=dst)
    router = FallbackRouter(cfg)
    ok = await router.execute()
    report = router.get_diagnostics_report()
    report["media_id"] = media_id
    if not ok:
        # push the structured report to your DLQ / alerting channel here
        logger.error("Final FAILED: %s", report["failure_classification"])
    return report

The same windowing and timeout discipline applies whether this runs as an FFmpeg batch processing for podcasts job or under a Celery task routing for video jobs worker — wrap execute() in a task that respects per-worker concurrency limits so the fallback pool never oversubscribes the CPU.

Verification

Confirm the router actually degrades rather than retrying blindly. First, force a CUDA_INIT failure on a GPU-less host or by hiding the driver, and watch the log emit the encoder swap:

CUDA_VISIBLE_DEVICES="" python -m worker.transcode sample.mov out.mp4
# Expected:
#   [INFO] Attempt 1/2 | Encoder: h264_nvenc
#   [WARNING] Failure detected: CUDA_INIT (exit 1). Preparing fallback routing.
#   [INFO] Routing to software fallback encoder: libx264
#   [INFO] Attempt 2/2 | Encoder: libx264
#   [INFO] Transcode succeeded on attempt 2.

Then verify the output the fallback produced is a valid, software-encoded H.264 file — the encoder tag should read a software build, not nvenc:

ffprobe -v error -select_streams v:0 \
  -show_entries stream=codec_name,profile,pix_fmt \
  -show_entries format_tags=encoder \
  -of default=noprint_wrappers=1 out.mp4
# codec_name=h264
# profile=High
# pix_fmt=yuv420p
# TAG:encoder=Lavc60.x libx264

Finally, assert the diagnostics contract in a test so a regression that breaks classification fails CI:

import asyncio

def test_fallback_records_cuda_init():
    cfg = TranscodeConfig(input_path="sample.mov", output_path="out.mp4")
    router = FallbackRouter(cfg)
    asyncio.run(router.execute())
    report = router.get_diagnostics_report()
    trace = report["execution_trace"]
    assert trace[0]["encoder"] == "h264_nvenc"
    assert trace[0]["failure_mode"] == "CUDA_INIT"
    assert trace[-1]["encoder"] == "libx264"        # proved it actually swapped

Failure Modes & Edge Cases

Edge case What happens Mitigation
Fallback also OOMs libx264 multi-pass on a 4K source exhausts host RAM, second attempt fails Cap worker RSS with a cgroup; lower -preset to veryfast and force single-pass on the fallback path
Benign NVENC warning misclassified A non-fatal nvenc warning matches CUDA_INIT and triggers a needless software run Keep stderr_window tight; require a non-zero exit and signature match before swapping
Partial output left on disk Primary attempt writes a truncated out.mp4 before failing; fallback -y overwrites it Always pass -y; write to a .tmp path and atomic-rename only on final_status == SUCCESS
Encoder-fault storm Many concurrent jobs hit CUDA_INIT/OOM_KILLED in one window Add a scheduler-level circuit breaker that routes all new jobs to software until driver contexts stabilize
10-bit HDR source h264_nvenc rejects the profile; libx264 succeeds but downconverts Detect pix_fmt upstream and pin -pix_fmt yuv420p or route to hevc_nvenc with a 10-bit profile
Audio-only job Video flags are irrelevant; failure surface is libfdk_aac vs aac Reuse the same router with audio encoders; this mirrors audio codec normalization workflows

FAQ

Why not just retry the same GPU encoder a few times?

Because the dominant failures here are deterministic, not transient. A lost CUDA context or a profile mismatch fails identically on every retry, so retrying on h264_nvenc only burns the max_retry_attempts budget and keeps the dead context in rotation. Retry is the right tool for transient I/O; encoder swapping is the right tool for a sticky hardware fault. The router does both — one retry, but on a different encoder.

Why classify against only the last 50 stderr lines?

FFmpeg emits thousands of progress and warning lines per job. Matching the full buffer means a benign warning early in a long encode can trigger a false CUDA_INIT match and pointlessly demote a healthy job to software. Bounding the match window to the tail — where the fatal error actually lands — makes classification both faster and far less prone to false positives. Pair it with the exit code: signature plus non-zero exit, never signature alone.

Should container corruption ever fall back to software?

No. A missing moov atom or truncated mdat fails on every encoder identically, so a software re-run wastes CPU and delays the inevitable. The router routes CONTAINER_DEADLOCK straight to quarantine and leaves structural validation to the pre-flight gate. Keeping that boundary clean is what lets the fallback router stay focused on compute-bound faults instead of becoming a catch-all error handler.