Reliable webhooks: dead-letter queues, retries, and replay
Webhooks are easy to send, hard to deliver reliably. Here is the pattern we use: exponential backoff, dead-letter queuing, and idempotent replay — so no event is ever silently dropped.
Why webhook delivery is hard
A webhook is a fire-and-forget HTTP POST. The hard part is what happens when the receiver is down, returns a 5xx, or times out. "Send and forget" is not a reliability strategy — it is a data loss strategy.
The standard approach: retry with exponential backoff. But naive retry adds new problems: idempotency (the receiver may process the same event twice), ordering (retried events arrive out of order), and observability (you do not know which events were delivered vs. dropped).
Our stack
We run webhooks as a background worker with three layers:
1. Outbox pattern. Every event is written to a webhook_deliveries table in the same transaction as the originating operation. The worker reads from this table. This guarantees that no event is lost due to a crash between the business logic and the HTTP send.
2. Exponential backoff with jitter. Failed deliveries are retried at 1s, 2s, 4s, 8s, 16s — with ±20% jitter to avoid thundering-herd on the receiver. After 6 attempts, the delivery is moved to a dead-letter queue (DLQ) with the full payload and error log.
3. Dead-letter queue + replay API. The DLQ is a webhook_dlq table with a replayed_at column. Tenants can replay DLQ entries via the dashboard or API — useful when a receiver was down for maintenance and needs to catch up. Replay uses the original idempotency key so the receiver can deduplicate safely.
Idempotency
Each webhook delivery carries an X-Idempotency-Key header derived from the event ID. Receivers that check this header can deduplicate replays without coordinating with us. We document the idempotency contract in the API docs — it shifts the responsibility appropriately.
What we monitor
DLQ depth is our primary webhook health metric. A growing DLQ means either receivers are down or our retry logic has a bug. We page on DLQ depth > 100 per tenant. Normal operation: DLQ is empty.
The result: in eight months of production, we have had zero silently-dropped webhook events. Every failure is visible, replayable, and auditable.