Reliable Webhook Delivery: Architecture for Outbound HTTP You Can Trust

You add a sendWebhook() call to your checkout flow. It fires a POST to the customer’s callback URL. Works on your machine. Works in staging. Ships.

Two weeks later, support tickets start arriving. A customer swears they received the same “payment succeeded” event seventeen times. Another says they never got the “subscription cancelled” event at all — but your logs show a 200 OK from their endpoint. A third customer’s server was down for maintenance; your retry loop hammered them with 500 req/s for twenty minutes until they threatened to block your IP range.

The problem is not the webhook payload. The problem is that you treated an unreliable, asynchronous, multi-tenant delivery channel like a synchronous function call inside a happy-path request handler. HTTP to a URL you do not control is not a method invocation. It is a message over a noisy, slow, occasionally broken pipe. You need an architecture that respects that.

This post builds one. A small but complete delivery pipeline: queue decoupling, exponential backoff with jitter, per-endpoint circuit breakers, a dead-letter queue for permanent failures, and the metrics that tell you whether your webhooks are a feature or a liability. All in TypeScript you can drop into an existing Node.js service.

The fire-and-forget trap

The naive implementation looks like this:

async function checkout(order: Order) {
  await db.orders.insert(order);
  await fetch(customer.webhookUrl, {
    method: 'POST',
    body: JSON.stringify({ event: 'order.created', order }),
    headers: { 'Content-Type': 'application/json' },
  });
}

Everything here is wrong for production. The fetch has no timeout, so a slow endpoint blocks the checkout response for thirty seconds or forever. If the endpoint returns a 500, the event is lost — no retry, no log, no trace. If your server restarts mid-flight, the event disappears with it. If the customer has two endpoints and one is down, the down one gets no retries and the up one gets duplicates on every checkout.

Worse, when you inevitably add retries, you add them inside the request handler:

for (let i = 0; i < 3; i++) {
  try {
    await fetch(url, { body, signal: AbortSignal.timeout(5000) });
    break;
  } catch { /* wait 1s and retry */ }
}

Now a flaky endpoint adds three to five seconds of latency to every checkout. The checkout handler is doing HTTP delivery work, so scaling checkout means scaling delivery, even though the two have completely different load patterns. And if the endpoint is down for an hour, every checkout generates three failed requests that clutter your logs and burn CPU.

The fix is separation of concerns. The checkout handler creates an event and writes it to a queue. A dedicated worker handles the messy business of HTTP delivery. The checkout finishes in milliseconds. The worker retries, backs off, and survives restarts.

Queue decoupling: the outbox or a job table

You do not need Redis to decouple. If you already use Postgres, a job table with SKIP LOCKED is enough. The pattern is simple: inside the same transaction that writes the business data, write a row to a webhook_deliveries table. A worker polls that table, locks rows with FOR UPDATE SKIP LOCKED, and processes them.

CREATE TABLE webhook_deliveries (
  id SERIAL PRIMARY KEY,
  subscription_id INT NOT NULL REFERENCES webhook_subscriptions(id),
  event_type TEXT NOT NULL,
  payload JSONB NOT NULL,
  attempts INT NOT NULL DEFAULT 0,
  next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  status TEXT NOT NULL DEFAULT 'pending',
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_webhook_deliveries_poll
ON webhook_deliveries (status, next_attempt_at)
WHERE status = 'pending';

The worker picks up pending rows whose next_attempt_at has passed:

const batch = await db.query(
  `SELECT id, subscription_id, event_type, payload, attempts
   FROM webhook_deliveries
   WHERE status = 'pending' AND next_attempt_at <= NOW()
   ORDER BY next_attempt_at
   LIMIT $1
   FOR UPDATE SKIP LOCKED`,
  [concurrency]
);

Why SKIP LOCKED? Because you will run multiple worker processes. Without it, two workers race for the same row and one blocks until the other commits. With it, the second worker simply skips the locked row and grabs the next available one. The blog has a full walkthrough of this pattern in the skip-locked post; the short version is that it gives you a fair, concurrent, restart-safe queue without adding infrastructure.

The crucial detail is that the event is written in the same database transaction as the business operation. If the checkout commits, the webhook row commits. If the checkout rolls back, the webhook row vanishes. You never send a webhook for data that was not actually persisted.

Retry with exponential backoff and jitter

A customer’s endpoint returning 502 Bad Gateway does not mean “give up.” It means “try again later, slowly.” The retry schedule that actually works is exponential backoff with full jitter. Not fixed delays. Not exponential without jitter — that causes thundering herds when an endpoint recovers and every queued event hits it simultaneously.

function delayMs(attempt: number, baseMs = 1000, maxMs = 60_000): number {
  const cap = Math.min(maxMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * cap);
}

Attempt 0: delay between 0 and 1 second.
Attempt 1: delay between 0 and 2 seconds.
Attempt 2: delay between 0 and 4 seconds.
Attempt 6: delay between 0 and 60 seconds (capped).

This spreads retries across time and eliminates synchronization. If the endpoint comes back at 14:03:22, the ten queued events hit it over the next minute instead of in the same second.

Set a maximum attempt count — ten is usually enough — and after the last failure, move the event to a dead-letter state. Do not retry forever. A permanent failure deserves human attention, not infinite CPU burn.

const MAX_ATTEMPTS = 10;

async function attemptDelivery(
  delivery: WebhookDelivery,
  subscription: Subscription
): Promise<'success' | 'retry' | 'dead'> {
  try {
    const res = await fetch(subscription.url, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'X-Webhook-Attempt': String(delivery.attempts + 1),
        'X-Webhook-Event-ID': delivery.id,
        'X-Webhook-Signature': signPayload(delivery.payload, subscription.secret),
      },
      body: JSON.stringify(delivery.payload),
      signal: AbortSignal.timeout(30_000),
    });

    if (res.status >= 200 && res.status < 300) return 'success';
    if (res.status >= 400 && res.status < 500) return 'dead'; // 4xx is permanent
    return 'retry';
  } catch (err) {
    if (err instanceof TypeError || err.name === 'AbortError') {
      // Network failure or timeout — definitely retry
      return 'retry';
    }
    return 'dead';
  }
}

Notice the 4xx handling. A 410 Gone or 404 Not Found means the endpoint does not want this event. Retrying a 404 is not resilience; it is harassment. Treat every 4xx as a terminal failure. Treat 5xx and network errors as retryable. Log the status code either way.

Circuit breaker per endpoint

Retries assume the endpoint is temporarily sick. But what if the endpoint is dead for an hour? You do not want every worker process trying it every few seconds, burning CPU and network, filling logs with the same failure. You want to stop trying for a while.

A circuit breaker tracks recent failures per endpoint. After a threshold of consecutive failures, it flips to OPEN and rejects further attempts for a cooldown period. After the cooldown, it moves to HALF_OPEN and allows a single probe. If the probe succeeds, it closes. If not, it opens again.

Because you have many customers with different endpoints, the breaker must be scoped per subscription URL, not global.

type State = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

class CircuitBreaker {
  private state: State = 'CLOSED';
  private failures = 0;
  private lastFailure = 0;
  private readonly threshold = 5;
  private readonly cooldownMs = 60_000;

  recordSuccess() {
    this.failures = 0;
    this.state = 'CLOSED';
  }

  recordFailure() {
    this.failures++;
    this.lastFailure = Date.now();
    if (this.failures >= this.threshold) {
      this.state = 'OPEN';
    }
  }

  canAttempt(): boolean {
    if (this.state === 'CLOSED') return true;
    if (this.state === 'HALF_OPEN') return true;
    if (this.state === 'OPEN') {
      if (Date.now() - this.lastFailure >= this.cooldownMs) {
        this.state = 'HALF_OPEN';
        return true;
      }
      return false;
    }
    return false;
  }
}

Store the breaker in memory keyed by subscription_id. If a worker restarts, the breaker resets — acceptable because the first few attempts will fail quickly and reopen it. For a distributed fleet, consider a shared Redis key with a short TTL, but in practice per-process breakers are usually enough.

In the worker, check the breaker before every delivery attempt:

const breaker = breakers.get(subscription.id) ?? new CircuitBreaker();
breakers.set(subscription.id, breaker);

if (!breaker.canAttempt()) {
  await postponeDelivery(delivery.id, Date.now() + 60_000);
  continue;
}

const result = await attemptDelivery(delivery, subscription);
if (result === 'success') breaker.recordSuccess();
else breaker.recordFailure();

Dead letter queue: when to give up

After MAX_ATTEMPTS failures, move the delivery to a terminal dead status and write the last error. Dead events live in the same table with a status filter — no need for a separate queue.

ALTER TABLE webhook_deliveries ADD COLUMN last_error TEXT;
ALTER TABLE webhook_deliveries ADD COLUMN dead_at TIMESTAMPTZ;

The worker update:

if (result === 'dead' || delivery.attempts + 1 >= MAX_ATTEMPTS) {
  await db.query(
    `UPDATE webhook_deliveries
     SET status = 'dead', dead_at = NOW(), last_error = $2
     WHERE id = $1`,
    [delivery.id, lastError?.message ?? 'max attempts exceeded']
  );
  // Alert: a webhook is permanently stuck
  metrics.increment('webhook.dead', { subscription_id: subscription.id });
} else {
  const nextDelay = delayMs(delivery.attempts + 1);
  await db.query(
    `UPDATE webhook_deliveries
     SET attempts = attempts + 1, next_attempt_at = NOW() + $2::interval
     WHERE id = $1`,
    [delivery.id, `${Math.round(nextDelay / 1000)} seconds`]
  );
}

Dead events are not garbage. They are a signal that a customer endpoint is misconfigured or permanently broken. Surface them in an admin dashboard. Offer a manual retry button that resets status to pending and attempts to zero. And if a subscription accumulates more than a hundred dead events in a day, consider pausing it automatically and emailing the customer.

Observability: per-endpoint metrics

A webhook system without metrics is a black box. You need at least three dimensions of visibility:

Queue depth — how many events are waiting right now. Alert if it grows faster than the workers drain it.

Delivery latency — time from next_attempt_at to a terminal status. Not just HTTP round-trip; queue wait time matters too.

Per-endpoint outcome — success count, retry count, dead count, and HTTP status codes, broken down by subscription_id (or a hashed identifier to avoid high-cardinality explosions in your metrics backend).

metrics.histogram('webhook.delivery_latency_ms', latency, {
  subscription_id: hash(subscription.id),
  event_type: delivery.event_type,
});

metrics.increment('webhook.outcome', {
  subscription_id: hash(subscription.id),
  result: result === 'success' ? 'success' : 'dead',
  http_status: String(statusCode ?? 'network_error'),
});

Log every delivery attempt with the event ID, subscription ID, attempt number, outcome, and latency. Use structured logs — the blog has a full post on structured logging with Pino — so you can search by event_id and see the complete retry history in one query.

The worker loop

Putting it together, the worker is a tight loop with concurrency control:

const CONCURRENCY = 20;
const breakers = new Map<number, CircuitBreaker>();

async function runWorker() {
  while (true) {
    const pending = await fetchPendingBatch(CONCURRENCY);
    if (pending.length === 0) {
      await sleep(1000);
      continue;
    }

    await Promise.all(pending.map(async (delivery) => {
      const subscription = await fetchSubscription(delivery.subscription_id);
      const breaker = breakers.get(subscription.id) ?? new CircuitBreaker();
      breakers.set(subscription.id, breaker);

      if (!breaker.canAttempt()) {
        await postponeDelivery(delivery.id, Date.now() + 60_000);
        return;
      }

      const start = performance.now();
      const result = await attemptDelivery(delivery, subscription);
      const latency = Math.round(performance.now() - start);

      if (result === 'success') {
        breaker.recordSuccess();
        await markDelivered(delivery.id);
      } else if (result === 'dead') {
        breaker.recordFailure();
        await markDead(delivery.id, 'terminal status or max attempts');
      } else {
        breaker.recordFailure();
        await scheduleRetry(delivery);
      }

      metrics.increment('webhook.outcome', { subscription_id: hash(subscription.id), result });
      metrics.histogram('webhook.delivery_latency_ms', latency, { subscription_id: hash(subscription.id) });
    }));
  }
}

Run multiple instances of this worker. Because SKIP LOCKED prevents double delivery, scaling is horizontal and safe. If one worker crashes mid-batch, the rows it locked are released after the connection times out and another worker picks them up.

One more thing: idempotency on the receiving side

Even with all of this, you will occasionally deliver the same event twice. A worker might crash after the HTTP response returns but before the database commit. Postgres FOR UPDATE is not a distributed lock across your process and the customer’s server. The only way to guarantee exactly-once delivery is coordination with the receiver.

Include an X-Webhook-Event-ID header — derived from the delivery row ID or a UUID generated at event creation — and document that receivers should deduplicate on it. If your customers are engineers, they will thank you. If they are not, they will at least have a clear identifier to show their support team when duplicates arrive.

Practical takeaway

The difference between a webhook feature and a webhook liability is not the payload format. It is the delivery pipeline: queue decoupling, exponential backoff with jitter, per-endpoint circuit breakers, a dead-letter state with observability, and a documented idempotency contract. Together these cost maybe two hundred lines of TypeScript and a few database columns. The alternative is angry customers, blocked IP ranges, and 3 a.m. pages about ” duplicate events.”

Do not fire and forget. Fire, track, back off, and know when to stop.

A note from Yojji

The plumbing that moves data from your system to someone else’s — reliable outbound HTTP, retry discipline, circuit breakers, and delivery observability — is the kind of backend work that separates a prototype from a production integration. It is also the kind of engineering Yojji’s teams build into the systems they ship for clients.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and API-heavy backends — including the webhook infrastructure that keeps third-party integrations from becoming support nightmares.