Polling vs Webhooks for Async Transcription

This page sits under Async Transcription Queue Management within the Transcription & Speaker Diarization pipeline, and it answers a design question every media backend hits the moment it stops transcribing synchronously: once a job is submitted to a hosted engine that takes minutes to finish, how do you collect the result? The two options — polling the job status on a timer, or receiving a webhook callback — have very different reliability, latency, cost, and security profiles, and the production answer is almost always “webhooks, with polling as a safety net.”

Problem Framing

A multi-minute transcription job cannot block a request thread, so submission and collection are decoupled. If you naively poll every second you burn rate limit and money on jobs that are nowhere near done; if you poll too slowly you add minutes of avoidable latency before the transcript reaches chapter generation. If you rely purely on webhooks you inherit the classic distributed-systems failure: the callback is fired exactly once, and if your endpoint was redeploying or briefly 503-ing when it arrived, the result is simply gone and the job sits “done on their side, never collected on yours.” The collection log shows the two opposite pathologies:

WARN  poll   job=t_9f3a status=processing polls=142 elapsed=71s  note="tight 0.5s loop, 429 rate-limited"
ERROR poll   job=t_9f3a http=429 retry_after=30  note="polling budget exhausted for account"
WARN  hook   job=t_7c11 event=received sig_valid=False  note="unsigned payload, rejected"
ERROR hook   job=t_5d02 event=missed  detected_by=reconcile_sweep age=1840s  note="callback never arrived"

The first two lines are a polling loop with no backoff eating the rate limit; the last two are webhook problems — one spoofed/unsigned payload correctly rejected, and one genuine result that never arrived and was only recovered when a reconciliation sweep noticed the job had been completed upstream for half an hour. A robust collector must therefore verify every callback, handle each result exactly once even under retries, and still guarantee eventual collection when a webhook is dropped.

Solution Architecture

Use webhooks as the primary, low-latency path and a slow polling sweep as the backstop. On submission you register a webhook URL and persist the job with a pending state and an idempotency key. When the callback arrives you verify its signature, look up the job, and process it exactly once — a second delivery of the same event is detected and dropped. Separately, a periodic reconciliation job polls the status of any job that has been pending longer than a threshold; if the engine reports it completed but you never collected it, the sweep fetches the transcript through the same idempotent handler the webhook uses. Both paths converge on one collection function, so a result handled twice is a no-op.

Webhook-primary collection with a polling safety net A submitter registers a transcription job with a webhook URL and persists it as pending with an idempotency key in the job store. The transcription engine, when finished, fires a signed webhook to the receiver, which verifies the signature, dedupes on the idempotency key, and calls the single collection handler that writes the transcript and marks the job done. In parallel, a reconciliation sweep periodically polls the engine for jobs that have been pending too long; when it finds a completed job that was never collected, it calls the same collection handler, so a dropped webhook is still recovered. Submitter webhook_url + key Job store state=pending idempotency_key Transcription engine hosted async API Webhook receiver verify signature dedupe on key Reconcile sweep poll pending > 5 min backoff, capped Collection handler write once · mark done → diarization stage signed callback poll status lookup / dedupe

The design choice is not polling versus webhooks in the abstract — it is which is primary. Webhooks win on latency and cost because you do zero work until there is a result; polling wins on reliability because it is pull-based and cannot be silently dropped. Combining them keeps the latency of webhooks and the guaranteed-delivery of polling.

Implementation

1. Submit with a webhook URL and idempotency key

On submission, register the callback and persist the job before the engine can possibly call back. Store an idempotency key you control so both collection paths can dedupe on it. The official AssemblyAI API documentation carries the current webhook field names and status values.

# python 3.10+  requests==2.32.3  redis==5.0.4
import hashlib, hmac, json, logging, os, time
import requests

logging.basicConfig(level=logging.INFO, format="%(levelname)s | %(message)s")
logger = logging.getLogger(__name__)

API_BASE = "https://api.assemblyai.com/v2"
HEADERS = {"authorization": os.environ["ASSEMBLYAI_API_KEY"]}
WEBHOOK_SECRET = os.environ["WEBHOOK_SECRET"]     # shared secret for HMAC verification

def submit_job(audio_url: str, job_store) -> str:
    """Submit a transcript job with a webhook callback and persist it as pending."""
    idem_key = hashlib.sha256(audio_url.encode()).hexdigest()[:32]
    payload = {
        "audio_url": audio_url,
        "webhook_url": "https://pipeline.example.com/hooks/transcript",
        # engine signs the callback with this header/value so we can verify it:
        "webhook_auth_header_name": "X-Pipeline-Signature",
        "webhook_auth_header_value": WEBHOOK_SECRET,
    }
    resp = requests.post(f"{API_BASE}/transcript", json=payload, headers=HEADERS, timeout=15)
    resp.raise_for_status()
    job_id = resp.json()["id"]
    job_store.hset(f"job:{job_id}", mapping={
        "state": "pending", "idem_key": idem_key,
        "submitted_at": int(time.time()), "audio_url": audio_url,
    })
    logger.info(f"Submitted job {job_id} (idem={idem_key})")
    return job_id

2. Verify the signature and dedupe on receipt

Never trust an inbound callback. Verify the signature with a constant-time compare, then dedupe: if this job is already done, acknowledge with 200 and do nothing, so the engine’s own retries never double-process a transcript.

def handle_webhook(request_body: bytes, header_sig: str, job_store) -> tuple[int, str]:
    """Verify, dedupe, and hand off exactly once. Returns (http_status, message)."""
    expected = hmac.new(WEBHOOK_SECRET.encode(), request_body, hashlib.sha256).hexdigest()
    # constant-time compare defeats timing attacks on the signature
    if not hmac.compare_digest(expected, header_sig or ""):
        logger.warning("Rejected webhook: signature mismatch")
        return 401, "invalid signature"

    event = json.loads(request_body)
    job_id, status = event["transcript_id"], event["status"]
    state = job_store.hget(f"job:{job_id}", "state")
    if state == b"done":
        logger.info(f"Duplicate webhook for {job_id}; already collected")
        return 200, "already processed"     # idempotent: retries are safe
    if status != "completed":
        job_store.hset(f"job:{job_id}", "state", status)   # e.g. 'error'
        return 200, f"noted status {status}"

    collect_result(job_id, job_store)       # single shared handler
    return 200, "collected"

3. Add the backoff polling sweep

The reconciliation job is the backstop for dropped webhooks. It polls only jobs stuck pending past a threshold, with capped exponential backoff so a slow engine never turns into a rate-limit storm, and it routes any newly-completed job through the same collect_result handler.

def reconcile_pending(job_store, max_age_s: int = 300):
    """Poll jobs pending longer than max_age_s; recover any completed-but-uncollected."""
    now = int(time.time())
    for key in job_store.scan_iter("job:*"):
        job = job_store.hgetall(key)
        if job.get(b"state") != b"pending":
            continue
        if now - int(job[b"submitted_at"]) < max_age_s:
            continue
        job_id = key.decode().split(":", 1)[1]
        delay = 1.0
        for attempt in range(5):            # capped backoff: 1,2,4,8,16s
            r = requests.get(f"{API_BASE}/transcript/{job_id}", headers=HEADERS, timeout=15)
            if r.status_code == 429:
                time.sleep(min(delay, 16)); delay *= 2; continue
            status = r.json().get("status")
            if status == "completed":
                logger.warning(f"Recovered dropped webhook for {job_id} via sweep")
                collect_result(job_id, job_store)
            break

The single collect_result function that both paths call is what makes the whole thing safe: it fetches the transcript, writes it, and marks the job done in one guarded step, so a webhook and a sweep racing on the same job produce one transcript, not two. Downstream, that collected transcript flows into the same async transcription queue management workers that fan the job out to diarization and chapter generation.

Verification

Prove reliability, latency, and idempotency separately before trusting the collector on live traffic.

# 1. Idempotency: replay the same signed webhook twice; the second must be a no-op.
SIG=$(python -c "import hmac,hashlib,os;print(hmac.new(os.environ['WEBHOOK_SECRET'].encode(),open('hook.json','rb').read(),hashlib.sha256).hexdigest())")
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8000/hooks/transcript \
  -H "X-Pipeline-Signature: $SIG" --data-binary @hook.json
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8000/hooks/transcript \
  -H "X-Pipeline-Signature: $SIG" --data-binary @hook.json
# expected: 200 then 200 ("already processed"); transcript written exactly once

# 2. Signature rejection: tamper one byte, expect 401.
curl -s -o /dev/null -w "%{http_code}\n" -X POST localhost:8000/hooks/transcript \
  -H "X-Pipeline-Signature: deadbeef" --data-binary @hook.json
# expected: 401
# 3. Dropped-webhook recovery: mark a completed job as 'pending', run the sweep,
#    and assert it gets collected.
from pipeline import reconcile_pending, job_store
job_store.hset("job:t_5d02", mapping={"state": "pending", "submitted_at": 0})
reconcile_pending(job_store, max_age_s=0)
assert job_store.hget("job:t_5d02", "state") == b"done"
print("recovered dropped webhook via sweep")

A healthy run shows the duplicate webhook returning 200 with the transcript written once, the tampered payload returning 401, and the reconciliation sweep flipping a stuck job to done. Track webhook-to-collection latency, sweep recovery counts, and 429 rates as metrics — a rising sweep-recovery count means webhooks are being dropped and your endpoint’s uptime, not your polling, needs attention.

Failure Modes & Edge Cases

Edge case Symptom Remediation
Tight polling loop, no backoff 429 rate-limit, polling budget exhausted Cap exponential backoff at 16 s; poll only jobs older than the threshold
Webhook arrives before job persisted Receiver can’t find the job, drops the result Persist as pending before submitting, or buffer unknown-job callbacks briefly
Unsigned or spoofed callback Arbitrary payload marks a job done Reject on hmac.compare_digest mismatch; never process an unverified body
Engine retries a delivered webhook Transcript written twice, chapters duplicated Dedupe on job state; return 200 when already done
Endpoint down during callback Result never collected, job stuck pending Reconciliation sweep recovers it; alert if recovery age is high
Sweep and webhook race same job Two collectors fetch simultaneously Guard collect_result with a state check-and-set so only one writes
Threshold too low on slow engine Sweep polls jobs still legitimately processing Set max_age_s above the engine’s p95 turnaround, not below it

For pipelines processing large back catalogues, keep the reconciliation sweep on a slow cadence — every minute or two is ample — and let webhooks carry the real-time load, so your steady-state polling cost stays near zero while the safety net still guarantees eventual collection.

FAQ

If webhooks are more efficient, why keep polling at all?

Because a webhook is a single push that can be silently lost — your endpoint redeploys, a load balancer drops the connection, or the callback 503s and the engine gives up retrying. Polling is pull-based, so it cannot be lost the same way. Running a slow reconciliation sweep over jobs that have been pending too long guarantees eventual collection without paying the per-second polling cost on every job.

How do I stop a retried webhook from processing a transcript twice?

Make collection idempotent and dedupe on state. Store each job with a key you control, and have the receiver check whether the job is already done before doing any work; if it is, return 200 and stop. Guard the shared collection handler with a check-and-set on job state so that even a webhook and a reconciliation sweep hitting the same job at once produce exactly one written transcript.

How do I verify a webhook actually came from the transcription engine?

Register a shared secret at submission and have the engine echo it in a signed header. On receipt, recompute an HMAC-SHA256 over the raw request body with that secret and compare it to the header value using a constant-time comparison such as hmac.compare_digest, which avoids leaking the match through timing. Reject anything that fails with a 401 and never parse the body of an unverified request as authoritative.