Request Timeouts and Deadline Propagation: Stop the Chain of Slowness

Your frontend times out after 30 seconds. The API gateway returns 504 Gateway Timeout. The user refreshes. Now you have two requests in flight, then four, then eight. Behind the gateway, the user service is still waiting on the billing service, which is still waiting on a Postgres query that acquired a row lock because the previous query is holding it. Nothing has crashed. CPU is low. But every layer is saturated with requests that are already doomed, and every retry makes it worse.

This is the chain of slowness. It starts with one missing timeout and spreads because no layer knows when to stop waiting.

The fix is not “add more replicas.” The fix is deadline propagation: every downstream call gets a shrinking time budget, and when the budget hits zero, everything in the chain stops immediately. This post shows how to implement it in Node.js from the HTTP boundary down to the database driver, with code you can copy into an existing service.

The default timeout behavior is a lie

Most Node.js services look like this:

const user = await fetchUser(userId);
const billing = await fetchBilling(userId);
const permissions = await fetchPermissions(userId);

Each of those functions probably has a fetch call with some default timeout — maybe 30 seconds, maybe none at all. The problem is not the value. The problem is that each call starts its own independent clock.

If the upstream request has already been waiting for 28 seconds when fetchBilling starts, and fetchBilling hits its 30-second timeout, the user has waited 58 seconds total. The gateway gave up at 30. The user’s browser retried twice. The backend processed three identical requests, each spawning more slow calls.

Without a shared deadline, timeouts are just random number generators that decide which error message the user sees.

What deadline propagation actually means

When a request enters your system, assign it a deadline: an absolute timestamp by which the response must be produced. Pass that deadline through every layer. Every downstream call uses remainingTime = deadline - now() as its timeout. If the remaining time is zero or negative, fail immediately without making the call.

The rules are simple:

1. Set the deadline at the edge. The API gateway or load balancer decides the total budget (e.g., 5 seconds).
2. Propagate it explicitly. Headers, context objects, or metadata — not ambient globals.
3. Subtract overhead. Serialization, network round-trip, and queueing all burn budget.
4. Respect it at every I/O boundary. HTTP clients, database drivers, message producers, cache lookups.

This sounds like infrastructure work. It is mostly middleware. The payoff is that a slow dependency becomes a fast, contained error instead of a metastasizing pileup.

Edge-layer deadline: Express middleware

Start by attaching a deadline to every incoming request. Use req as the carrier.

const DEFAULT_DEADLINE_MS = 5_000;

export function deadlineMiddleware(req, res, next) {
  const clientTimeout = Number(req.header('x-request-timeout-ms'));
  const budget = clientTimeout > 0
    ? Math.min(clientTimeout, DEFAULT_DEADLINE_MS)
    : DEFAULT_DEADLINE_MS;

  req.context = {
    ...req.context,
    deadline: Date.now() + budget,
  };

  next();
}

For gRPC or internal services, propagate the deadline in metadata. For HTTP, use a standard header. X-Request-Deadline or X-Request-Timeout-At are common. The key is that downstream services parse the same header and apply the same math. If your stack is homogeneous, a context object passed through async hooks or explicit function arguments is even better because headers get parsed at every hop.

Also set the HTTP response timeout on the server so the framework itself stops waiting:

const server = app.listen(PORT);
server.timeout = DEFAULT_DEADLINE_MS + 1_000; // graceful margin
server.keepAliveTimeout = 5_000;

This prevents Node from holding sockets open for the default 2 minutes when a handler is already past its useful life.

The timeout helper every service needs

Every downstream call should use a single helper that computes the remaining budget and applies it. Here is the one I copy into services:

export function remainingMs(ctx) {
  if (!ctx?.deadline) return 0;
  return Math.max(0, ctx.deadline - Date.now());
}

export async function fetchWithDeadline(url, options = {}, ctx) {
  const budget = remainingMs(ctx);
  if (budget <= 0) {
    const err = new Error('deadline exceeded before request');
    err.code = 'DEADLINE_EXCEEDED';
    throw err;
  }

  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), budget);

  try {
    const res = await fetch(url, {
      ...options,
      signal: controller.signal,
      headers: {
        ...options.headers,
        'x-request-deadline': String(ctx.deadline),
      },
    });
    return res;
  } finally {
    clearTimeout(timeoutId);
  }
}

Three details matter here.

AbortController instead of a fetch timeout option. Node’s fetch supports signal natively. The signal aborts the underlying socket, which frees TCP connections and stops bytes from flowing. A wrapper-level rejection does not close the socket; it just stops waiting.

Propagate the absolute deadline, not the remaining duration. If you send timeout: 1200ms, and the network takes 300ms to deliver it, the receiver starts with 1200ms even though 300ms already burned. Sending the absolute timestamp lets every hop compute its own truthful remaining time.

Fail fast when budget is already zero. Do not start a request that is already late. The error is the same as a mid-flight timeout — DEADLINE_EXCEEDED — but it avoids consuming a connection and potentially triggering a retry storm.

Database queries need deadlines too

The HTTP client is not the only place requests wait. Postgres queries hold connections and locks. A query that runs for 30 seconds because it is missing an index is just as damaging as a slow HTTP call.

Most Postgres drivers accept a query timeout. In pg with node-postgres:

import { Client } from 'pg';

export async function queryWithDeadline(client, sql, params, ctx) {
  const budget = remainingMs(ctx);
  if (budget <= 0) {
    const err = new Error('deadline exceeded');
    err.code = 'DEADLINE_EXCEEDED';
    throw err;
  }

  // pg query timeout is in milliseconds
  const result = await client.query({
    text: sql,
    values: params,
    query_timeout: budget,
  });

  return result;
}

query_timeout cancels the query on the server side with a statement_timeout error. This is critical: a client-side timeout alone kills the request in Node but leaves the query running in Postgres, consuming a connection and holding locks. Server-side timeout releases both.

For higher-level ORMs, find the equivalent option. Prisma uses $queryRaw with a transaction timeout. Sequelize has query options. Drizzle runs on pg and passes through the same config. If your ORM does not support query-level timeouts, wrap the call in your own Promise.race against a setTimeout that issues pg_cancel_backend — or switch to a driver that does.

Async jobs: the deadline leaks here most often

Background workers are where deadline discipline usually dies. A job queue entry has no HTTP request, no browser timeout, no gateway. It sits in a queue for any amount of time, then executes. If the job calls three services and each has a 30-second default timeout, one slow dependency turns a 500ms job into a 90-second job that blocks the worker and fills the queue.

Enqueue jobs with a deadline derived from the originating request, or with a job-specific SLA:

export async function enqueueEmailJob(userId, ctx) {
  const budget = remainingMs(ctx);
  // If the request is already nearly dead, do not enqueue more work.
  if (budget < 500) {
    throw new Error('insufficient budget to enqueue follow-up work');
  }

  await queue.add('send-email', {
    userId,
    deadline: Date.now() + Math.min(budget, 10_000),
  });
}

Then the worker checks the budget before every external call:

queue.process('send-email', async (job) => {
  const ctx = { deadline: job.data.deadline };

  const user = await fetchWithDeadline(
    `${USER_SERVICE}/users/${job.data.userId}`,
    {},
    ctx,
  );

  const template = await queryWithDeadline(
    dbClient,
    'SELECT * FROM email_templates WHERE id = $1',
    [job.data.templateId],
    ctx,
  );

  await emailProvider.send(user.email, template.body);
});

If the job’s deadline passes during execution, every subsequent call throws DEADLINE_EXCEEDED immediately. The worker moves on to the next job instead of waiting pointlessly.

Some teams object: “But the email is important! We should retry until it sends.” Yes — but that retry belongs in a separate dead-letter or scheduled-retry queue with its own SLA, not in the same worker thread that is blocking real-time work. The deadline protects the worker’s throughput. The retry policy protects the business outcome. They are separate concerns.

Handling DEADLINE_EXCEEDED correctly

When a downstream call throws because the deadline passed, the handler has three choices:

1. Fail the whole request. Return 504 or 503 to the caller. This is correct for endpoints where partial data is useless.
2. Return partial results. If the dashboard can show user info without billing status, degrade gracefully. Log the missing piece.
3. Retry internally with a smaller scope. Only if the operation is idempotent and the retry uses a fresh, shorter deadline. Never loop retries with the same deadline.

Do not convert DEADLINE_EXCEEDED into a generic 500 Internal Server Error. The status code tells the caller (and your dashboards) that the service gave up because time ran out, not because of a code bug. That distinction matters for alerting: 500 spikes mean deploy a fix; 504 spikes mean scale a dependency or tune a timeout.

app.get('/api/dashboard/:userId', async (req, res) => {
  const ctx = req.context;

  try {
    const [user, billing] = await Promise.all([
      fetchWithDeadline(`${USER_SERVICE}/users/${req.params.userId}`, {}, ctx),
      fetchWithDeadline(`${BILLING_SERVICE}/accounts/${req.params.userId}`, {}, ctx),
    ]);

    res.json({ user: await user.json(), billing: await billing.json() });
  } catch (err) {
    if (err.code === 'DEADLINE_EXCEEDED' || err.name === 'AbortError') {
      return res.status(504).json({ error: 'upstream_timeout', retryable: true });
    }
    throw err;
  }
});

Note the retryable: true flag in the response body. This tells well-behaved clients to back off instead of immediately retrying. Without it, the client retries instantly and amplifies the overload.

Deadlines and retries: never retry on the same budget

The retries post covers jittered backoff and budgets. There is one rule specific to deadlines: every retry attempt gets its own fresh deadline, computed from the original deadline, not from the previous attempt’s remaining time.

export async function fetchWithRetry(url, options, ctx, maxAttempts = 3) {
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await fetchWithDeadline(url, options, ctx);
    } catch (err) {
      const isLast = attempt === maxAttempts;
      const isTimeout = err.code === 'DEADLINE_EXCEEDED' || err.name === 'AbortError';

      if (isLast) throw err;
      if (isTimeout) throw err; // do not burn retries on timeouts

      const backoff = Math.min(100 * 2 ** attempt, 2_000);
      await new Promise((r) => setTimeout(r, backoff + Math.random() * 100));
    }
  }
}

If a call fails because the deadline was exceeded, do not retry. The budget is already gone. Retrying would just stack more late requests behind the first one. Retry only on transient errors that might succeed faster the next time: connection refused, 502, 503.

What to monitor

Deadlines produce four metrics that tell you where the system is actually slow:

• deadline_exceeded_total counter, labeled by layer (http_client, db_query, cache_lookup).
• deadline_remaining_ms histogram, recorded at the start of every downstream call. If p50 remaining time at the DB layer is 200ms, your total budget is too tight or your HTTP calls are too slow.
• request_duration_ms histogram, labeled by status and result (success, timeout, error).
• downstream_timeout_rate gauge per dependency. If the billing service’s timeout rate jumps from 0.1% to 5%, the problem is there, not in your code.

Alert on trends, not absolute counts. A sudden spike in deadline_exceeded_total for db_query while HTTP client timeouts stay flat means the database needs attention. A uniform spike across all layers means your edge timeout is too short for current traffic patterns.

The checklist for existing services

You don’t need to rebuild everything. This is a migration path:

1. Add deadlineMiddleware at the edge. Start with a generous budget (e.g., 10 seconds).
2. Replace raw fetch calls with fetchWithDeadline in one service at a time.
3. Add query_timeout to every database call through a wrapper.
4. Find every setTimeout and Promise.race hack that pretends to be a timeout and replace it with propagated deadlines.
5. Tighten the edge budget gradually as you observe p99 latencies.
6. Add the four metrics above.

The biggest resistance is usually “but we don’t know how long things take.” That is exactly why you need deadlines — because without them, you don’t know, and the first time you find out is at 3 a.m. when the queue is full.

Closing

A distributed system without deadline propagation is a system where every component makes optimistic guesses about how long it has to live. Those guesses compound. One slow query becomes twenty open connections, four retried requests, and a queue that takes twenty minutes to drain after the original problem is already fixed.

Wire deadlines through your stack: absolute timestamps at the edge, remainingMs at every downstream call, AbortController for HTTP, query_timeout for Postgres, and explicit job budgets for workers. Fail fast, return clear status codes, and reserve retries for errors that might actually succeed. The result is not just faster failures — it is a system that degrades predictably instead of collapsing suddenly.

A note from Yojji

Deadline propagation and graceful degradation under load are precisely the kinds of backend reliability concerns that separate a prototype from a production-grade distributed system. Yojji engineers regularly implement these patterns in high-throughput Node.js and cloud-native services, treating timeout discipline and failure isolation as non-negotiable baselines for systems that handle real traffic.