Celery vs Airflow for Video Encoding Queues

This page sits beneath Choosing a Media Pipeline Orchestrator within the Pipeline Automation & Batch Processing layer, and it answers one narrow question the parent decision framework raises: for a video encoding queue specifically — hundreds of independent, long-running ffmpeg jobs that need priority ordering and GPU affinity — do you reach for Celery or Apache Airflow? The general axes matter, but an encoding queue has a distinctive shape that pushes the answer harder in one direction than the general case suggests.

Problem Framing

A video encoding queue is not a DAG of dependent steps. It is a firehose of mutually independent units of work: encode ep0007 at 1080p, encode ep0007 at 720p, encode ep0119 at 480p, re-encode 4,000 back-catalogue files overnight. Each unit is long — a single H.264 or HEVC pass can run 10 to 45 minutes — and each competes for a scarce, expensive resource: the GPU. Two requirements dominate everything else:

  • Priority. A newly-published episode’s encodes must jump ahead of a bulk back-catalogue re-encode already in flight. Without priority, the marketing team waits behind 4,000 archive jobs.
  • GPU affinity. NVENC encodes must land on GPU workers; CPU-only work (packaging, loudness, thumbnail sheets) must not steal a GPU slot. Mixing them wastes VRAM and starves the queue.

Get the tool wrong and the symptoms are unmistakable. On Airflow, a busy encoding queue produces log lines like this:

[scheduler] WARNING - Task encode_1080p held slot for 2612s; pool 'gpu' at 4/4, 3140 tasks queued
[scheduler] ERROR   - Executor reports task instance finished (failed) although the task says it's running
[worker]    WARNING - heartbeat 91s late; marking encode_hevc as zombie

That last line is the trap: the encode did not fail — the worker was pegged running ffmpeg and missed its heartbeat, so the scheduler declared a healthy 40-minute job a zombie and retried it, doubling GPU cost. The encoding queue is stressing exactly the property Airflow is weakest on.

Solution Architecture

The clean answer for the queue itself is Celery: a worker-queue is the native model for many independent long jobs with priority and affinity. Airflow’s value does not disappear, though — it owns the calendar and backfill around the queue. The production architecture uses both, with a strict division of labor: Airflow decides what and when, Celery decides where and in what order it runs. The diagram shows that split.

Hybrid encoding queue: Airflow and event triggers feed Celery GPU-affinity priority queues On the left, an upload webhook and an Airflow scheduler both enqueue jobs. In the center, a Celery broker on Redis holds three priority queues: encode.gpu.high at priority nine fed by the webhook, encode.gpu.normal at priority five fed by Airflow backfills, and encode.cpu at priority three for packaging and loudness. On the right, the two GPU-affinity queues drain into a GPU worker pool configured with concurrency equal to the GPU count, late acknowledgement, and prefetch one, while the CPU queue drains into a separate CPU worker pool for packaging and loudness. Upload webhook new episode · event Airflow scheduler cron + backfill triggers, never hosts Celery broker (Redis) encode.gpu.high priority 9 · fresh episodes encode.gpu.normal priority 5 · backfills encode.cpu priority 3 · packaging, loudness GPU worker pool concurrency = GPU count acks_late · prefetch = 1 NVENC affinity CPU worker pool no GPU · oversubscribe OK

Read the split precisely. Airflow never runs ffmpeg inside a task slot — it submits the job to a Celery queue and returns immediately (or defers). Celery owns the priority ordering and the GPU/CPU affinity that the encoding queue actually needs. This is the concrete instance of the hybrid that the parent decision framework points at.

Implementation

When Celery wins — the priority queue with GPU affinity

For the queue itself, Celery is the direct expression of the requirement. Route by queue name to get affinity, and set task priority to get ordering. This extends the routing detailed in Celery task routing for video jobs.

# celery==5.4.0  (Redis broker; priority requires the Redis priority transport)
from celery import Celery
from kombu import Queue

app = Celery("encoding", broker="redis://localhost:6379/0")
app.conf.task_queues = (
    Queue("encode.gpu.high",   queue_arguments={"x-max-priority": 10}),
    Queue("encode.gpu.normal", queue_arguments={"x-max-priority": 10}),
    Queue("encode.cpu"),
)
app.conf.worker_prefetch_multiplier = 1   # never let one worker hoard long jobs
app.conf.task_acks_late = True            # a 45-min encode survives a restart

@app.task(bind=True, acks_late=True, max_retries=3,
          autoretry_for=(IOError,), retry_backoff=10)
def encode_video(self, src_uri: str, profile: str, codec: str = "h264_nvenc") -> dict:
    dst_uri = run_ffmpeg(src_uri, profile, codec)   # returns an s3:// URI
    return {"profile": profile, "uri": dst_uri, "sha256": digest_of(dst_uri)}

def enqueue_fresh(src_uri: str, profile: str):
    # fresh episode: high-affinity GPU queue, top priority
    return encode_video.apply_async(
        args=[src_uri, profile], queue="encode.gpu.high", priority=9)

def enqueue_backfill(src_uri: str, profile: str):
    return encode_video.apply_async(
        args=[src_uri, profile], queue="encode.gpu.normal", priority=5)

Start GPU workers bound to the affinity queues with concurrency equal to the GPU count, and CPU workers on the CPU queue:

# GPU host: one slot per physical GPU, both GPU queues, high drained first
CUDA_VISIBLE_DEVICES=0,1 celery -A encoding worker \
  -Q encode.gpu.high,encode.gpu.normal --concurrency 2 -n gpu@%h

# CPU host: packaging/loudness, safe to oversubscribe
celery -A encoding worker -Q encode.cpu --concurrency 8 -n cpu@%h

This gives you everything the encoding queue demands: fresh episodes preempt the backfill via priority=9 on the high queue, GPU work never lands on a CPU worker because affinity is enforced by queue subscription, and prefetch_multiplier=1 stops a single worker from grabbing ten 40-minute jobs and starving its peers. The authoritative flags live in the Celery routing and worker documentation.

When Airflow wins — the calendar and the backfill

Airflow’s strength is the part Celery is weak on: time-driven triggers, catchup, and one-command backfills over date ranges. The idiomatic pattern has Airflow submit to Celery rather than host the encode, so a long job never occupies a scheduler slot. This complements orchestrating pipelines with Airflow.

# apache-airflow==2.9.3  celery==5.4.0
from airflow.decorators import dag, task
import pendulum
from encoding import enqueue_backfill   # the Celery submitter above

@dag(schedule="0 2 * * *", start_date=pendulum.datetime(2025, 5, 1),
     catchup=True, max_active_tasks=8)      # backfill missed nights
def nightly_reencode():

    @task
    def discover() -> list[dict]:
        return find_stale_renditions()      # [{src, profile}, ...]

    @task
    def submit(job: dict) -> str:
        # hand off to Celery and return the task id — do NOT run ffmpeg here
        async_result = enqueue_backfill(job["src"], job["profile"])
        return async_result.id

    submit.expand(job=discover())

nightly_reencode()

The division is now literal: Airflow’s catchup=True and airflow dags backfill give you calendar recovery no Celery-only stack has, while the encode itself runs on a Celery GPU worker where priority and affinity apply. Airflow’s task finishes in milliseconds — it only enqueues — so the zombie-heartbeat failure mode from the problem framing never occurs.

Verification

Prove priority ordering and affinity hold under a real backlog before trusting the split.

# 1. Flood the normal queue with a backfill, then drop one high-priority job in.
python -c "from encoding import enqueue_backfill as b; [b('s3://arc/%d.mov'%i,'720p') for i in range(500)]"
python -c "from encoding import enqueue_fresh as f; f('s3://uploads/hot.mov','1080p')"

# 2. The fresh 1080p job must be picked up before the 500 backfill jobs drain.
celery -A encoding events --dump | grep -m1 "hot.mov"   # should appear near-immediately

# 3. Confirm GPU affinity: the CPU worker must never receive an encode task.
celery -A encoding inspect active_queues -d cpu@$(hostname) | grep -c encode.gpu   # expect 0

# 4. Confirm Airflow only enqueues (task duration is sub-second, not sub-hour).
airflow tasks states-for-dag-run nightly_reencode <run_id> | awk '{print $NF}'

A healthy split shows the hot 1080p job jumping the 500-job backfill, zero GPU tasks reaching the CPU worker, and Airflow submit tasks completing in well under a second.

Failure Modes & Edge Cases

Edge case Symptom Remediation
ffmpeg encode hosted inside an Airflow task Zombie/heartbeat retries, doubled GPU cost Submit to Celery and return the task id; never run the encode in a slot
worker_prefetch_multiplier left at default One worker hoards long jobs, others idle Set to 1 for long-running encode tasks
Priority ignored on RabbitMQ without x-max-priority Fresh episodes stuck behind backfill Declare queue_arguments={"x-max-priority": 10}, or use Redis priority transport
acks_late=False on a long encode Worker restart loses in-flight 40-min job Enable acks_late and set broker visibility timeout > job ceiling
GPU and CPU work share one queue NVENC jobs land on CPU-only hosts, VRAM idle Split into encode.gpu.* and encode.cpu; bind workers by -Q
Airflow catchup=True on first deploy Months of backfill stampede the GPU queue Set a recent start_date or catchup=False for the initial run
Retry re-encodes a completed rendition Wasted GPU minutes on idempotent work Key the task on sha256(src)+profile; short-circuit if dst_uri exists

FAQ

If Celery owns the queue, why involve Airflow at all?

Because Celery has no real calendar. Nightly re-encodes, weekly archive sweeps, and recovering a run that was missed during an outage are exactly what Airflow's schedule, catchup, and backfill give you for free. The rule is that Airflow triggers and backfills while Celery executes — Airflow submits a job to a Celery queue and returns immediately, so the long encode never occupies a scheduler slot and you still get one-command date-range recovery.

Can Airflow alone run a video encoding queue if I keep tasks short?

Only by making Airflow a worse Celery. You would need deferrable operators to avoid holding slots, an external executor for the actual ffmpeg process, and custom priority weighting to preempt a backfill — which is most of what Celery gives you natively. For a pure firehose of independent long encodes with priority and GPU affinity, Celery is the direct fit; reach for Airflow when a real calendar or a multi-stage dependency graph is the point.

How do I stop a backfill from starving fresh episode encodes?

Two mechanisms together. Route fresh episodes to a dedicated high-priority queue (encode.gpu.high, priority=9) that workers drain before the backfill queue, and set worker_prefetch_multiplier=1 so no worker pre-grabs a stack of backfill jobs it would have to finish first. With both in place, a newly published episode is picked up on the next free GPU slot regardless of how deep the archive backlog is.