Request Hedging: Cut Tail Latency In Half Without Overprovisioning

Your search service has ten replicas behind a load balancer. The median response time is 80 ms. The P95 is 150 ms. The P99 is 2.3 seconds. You have already ruled out slow queries, bad indexes, and GC pauses. The problem is one replica. Every minute, one pod gets a noisy neighbor on its VM, a brief CPU throttle, or a network blip, and every request routed to that pod pays the tax. The other nine pods are fine.

This is tail latency, and it is structural. The more hops a request takes through microservices, the worse it gets. If you call three services in parallel and each has a 1% chance of hitting a slow replica, your end-to-end P99 is now roughly 3%. Add five services and your P99 is ugly even when every individual service looks healthy.

Request hedging is the pattern that says: “If a request has not come back by the 95th-percentile time, send a duplicate to a different replica and take whichever arrives first.” It costs a small amount of extra load. It cuts your tail latency dramatically. And most teams do not use it because the naive implementation doubles your request volume and creates cancellation bugs.

This post is the safe implementation. The math for when it pays off. And the three guardrails that keep it from becoming a denial-of-service tool against your own service.

What hedging is, and what it is not

Hedging is not retrying. A retry waits for a failure (timeout, 5xx, connection error) and then sends a new request. Hedging does not wait for failure. It waits for a delay, then fires a second request in parallel while the first is still in flight. Whichever response arrives first wins. The loser is cancelled if possible.

This matters for read-heavy, latency-sensitive endpoints: search, recommendations, feature flags, user profile lookups, config reads. It does not work for mutations unless they are perfectly idempotent and your system can safely execute the same write twice. Do not hedge POST /charge.

The naive implementation and why it explodes

The first attempt most engineers write looks like this:

async function naiveHedgedFetch(url: string, delayMs: number) {
  const timer = new Promise((_, reject) =>
    setTimeout(() => reject(new Error('hedge')), delayMs)
  );
  return Promise.race([fetch(url), timer]).catch(() => fetch(url));
}

This is dangerous for four reasons.

1. The first request is never cancelled. It keeps running on the target replica, consuming CPU, memory, and a connection slot, even though you already got the answer from the second request. You doubled the load on the slow replica instead of relieving it.

2. No hedging delay tuning. If you hedge at the median (80 ms), you send two requests for most calls. That is nearly 2x load for a marginal latency win.

3. No in-flight limit. If your service is already struggling, hedging sends more requests into the struggle. You turn a slow replica incident into a full cluster overload.

4. No replica diversity. If both requests hit the same overloaded pod (sticky sessions, hash-based routing, or bad load balancing), hedging does nothing.

The safe version fixes all four.

Safe hedging in Node.js

Here is the practical version. It uses AbortController to cancel the loser, enforces a maximum hedging ratio, skips hedging when downstream is already unhealthy, and picks the hedging delay from a real percentile.

interface HedgingOptions {
  hedgeDelayMs: number;       // e.g. 150 — your P95 or slightly above
  maxConcurrentHedges: number; // e.g. 20 — hard limit on in-flight hedges
  hedgeRateLimit: number;      // e.g. 0.10 — max 10% of requests hedged
}

interface HedgedResult<T> {
  winner: 'original' | 'hedge';
  result: T;
  loserCancelled: boolean;
}

class HedgingClient {
  private activeHedges = 0;
  private totalRequests = 0;
  private hedgedRequests = 0;

  constructor(private readonly opts: HedgingOptions) {}

  async fetch<T>(
    makeRequest: (signal: AbortSignal) => Promise<T>,
    isHealthy: () => boolean = () => true
  ): Promise<HedgedResult<T>> {
    this.totalRequests++;

    const shouldHedge =
      isHealthy() &&
      this.activeHedges < this.opts.maxConcurrentHedges &&
      this.hedgedRequests / this.totalRequests < this.opts.hedgeRateLimit;

    if (!shouldHedge) {
      const result = await makeRequest(new AbortController().signal);
      return { winner: 'original', result, loserCancelled: false };
    }

    const controller = new AbortController();
    let hedgeTimer: NodeJS.Timeout | null = null;
    let hedgeStarted = false;

    const original = makeRequest(controller.signal);

    const hedgePromise = new Promise<T>((resolve, reject) => {
      hedgeTimer = setTimeout(() => {
        if (controller.signal.aborted) return;
        hedgeStarted = true;
        this.activeHedges++;
        this.hedgedRequests++;

        const hedgeController = new AbortController();
        makeRequest(hedgeController.signal)
          .then((value) => {
            if (!controller.signal.aborted) {
              controller.abort(); // cancel the original
              resolve(value);
            }
          })
          .catch((err) => {
            if (!controller.signal.aborted) {
              reject(err);
            }
          })
          .finally(() => {
            this.activeHedges--;
          });
      }, this.opts.hedgeDelayMs);
    });

    return Promise.race([
      original.then((value) => {
        if (hedgeTimer) clearTimeout(hedgeTimer);
        if (hedgeStarted) this.activeHedges--;
        return { winner: 'original' as const, result: value, loserCancelled: hedgeStarted };
      }),
      hedgePromise.then((value) => {
        return { winner: 'hedge' as const, result: value, loserCancelled: true };
      }),
    ]).catch(async (err) => {
      if (hedgeTimer) clearTimeout(hedgeTimer);
      if (hedgeStarted) this.activeHedges--;
      throw err;
    });
  }
}

Key details in this implementation:

• Cancellation. The winner aborts the loser via AbortController. Your makeRequest must forward the signal to fetch, http.request, or whatever client you use. The losing request should error with an AbortError and free its socket.
• Delay placed at P95, not median. If your P95 is 150 ms, hedging at 150 ms means only ~5% of requests ever spawn a second call. The extra load is roughly 5%, not 100%.
• Hard concurrency cap. maxConcurrentHedges prevents a hedge storm during an incident. If 50 requests are already hedged, new requests go single-shot.
• Rate limiter over time. hedgeRateLimit is a safety valve. If you accidentally misconfigured the delay too low, the ratio guard caps the damage.
• Health gate. The isHealthy callback lets you wire in a circuit breaker or a custom “downstream is struggling” signal. When downstream is red, stop hedging. You do not send more traffic into a fire.

Wiring it into your HTTP client

Most teams should not write the above from scratch in every service. Wrap your internal HTTP client once:

import { HedgingClient } from './hedging';

const hedging = new HedgingClient({
  hedgeDelayMs: 150,
  maxConcurrentHedges: 20,
  hedgeRateLimit: 0.10,
});

export async function searchProducts(query: string) {
  const { result } = await hedging.fetch(
    (signal) =>
      fetch(`https://search.internal/products?q=${encodeURIComponent(query)}`, {
        signal,
      }).then((r) => r.json()),
    () => circuitBreakerForSearch.isClosed() && loadShedder.allows()
  );
  return result;
}

Notice the isHealthy lambda checks both the circuit breaker and a load shedder. This is the pattern that keeps hedging safe: you only hedge when you are confident the downstream pool can absorb the extra request.

The math: when hedging pays off

Hedging is not free. It adds load. The break-even depends on your latency distribution.

Suppose your latency CDF looks like this:

Percentile	Latency
P50	80 ms
P90	120 ms
P95	150 ms
P99	2300 ms

If you hedge at 150 ms, 5% of requests spawn a duplicate. Expected extra load is roughly 5% (ignoring cancellation delays). The P99 drops from 2300 ms to roughly 150 ms plus the second-request tail, which is usually around the P95 again. Call it 300 ms in the worst case.

You traded 5% more load for a P99 that dropped from 2.3 s to 300 ms. That is usually an enormous win for user-facing latency.

But if your latency distribution is smooth (P95 150 ms, P99 170 ms), hedging at 150 ms adds 5% load and your P99 only improves to 170 ms anyway. The gain is not worth the cost. Hedging pays off when the tail is fat — when the slowest 1% is an order of magnitude worse than the P95.

How to decide: plot a histogram of your endpoint latency. If the 99th percentile is more than 3x the 95th, hedging is probably worth it. If the 99th is less than 2x the 95th, fix your outliers first (noisy neighbors, GC tuning, better load balancing) before adding hedging.

Guardrails you must have in production

Hedging without guardrails is a load multiplier with a random delay. Add these before shipping.

1. Only hedge idempotent reads. Never hedge a charge, a state transition, or an email send. If executing twice is unsafe, do not hedge.

2. Cancel the loser aggressively. If your HTTP client does not respect AbortController, fix that first. The loser must close its TCP connection and free the slot. Node.js fetch (undici) does this correctly. Old request libraries may not.

3. Route hedges to a different replica. If your load balancer uses consistent hashing by user ID, the original and the hedge may land on the same pod. Use a different upstream URL, a different service discovery lookup, or add a hedge=true header that the load balancer uses to pick a different backend. Some teams run two identical internal DNS records and round-robin between them for hedges.

4. Cap the hedge ratio and concurrency. The code above has both. You need both. The ratio prevents gradual config drift from increasing load over weeks. The concurrency cap prevents a thundering herd during recovery.

5. Alert on hedge rate, not just latency. Emit two metrics: hedges_fired and hedges_won. If hedges_won is low, your delay is too short (you are hedging requests that would have finished soon anyway) or your replicas are all slow (hedging does not help if every pod is broken). If hedges_fired spikes, your tail latency got worse and you should investigate the root cause, not just enjoy the workaround.

When hedging is the wrong tool

Three cases where hedging makes things worse.

The whole fleet is slow. Hedging only helps when one replica is an outlier. If every pod is at 90% CPU, sending two requests means both take twice as long. You need horizontal scaling or query optimization, not hedging.

Your client does not support cancellation. If the losing request runs to completion on the server no matter what, you doubled the load on the slow replica. That is the opposite of what you want. Fix your client or your server middleware to support early aborts before hedging.

You are already at capacity limits. If your downstream has a strict rate limit (e.g., a third-party API with 100 req/s), hedging consumes two of those slots. You will hit the quota faster and get 429s for everyone. Use a local cache or a bulkhead instead.

The takeaway

Tail latency is not a tuning problem on the slow replica. It is a statistical inevitability in any large-enough fleet. Request hedging accepts that inevitability and works around it by betting that a second replica will be faster than the one outlier.

The safe version is small: delay at the P95, abort the loser, cap concurrent hedges, cap the hedge ratio, and skip hedging when downstream is unhealthy. It costs roughly 5% extra load for a P99 that can drop by an order of magnitude. For read-heavy, latency-sensitive microservices, that is one of the best load-to-latency trades you can make.

Measure your latency histogram first. If the tail is fat, add hedging with the guardrails above. If the tail is thin, fix the outliers. Either way, stop accepting 2-second P99s on a service whose healthy pods answer in 80 ms.

---

A note from Yojji

The kind of tail-latency engineering that turns a 2-second P99 into a 300 ms P99, request hedging, circuit breakers, and load-aware clients, is the kind of distributed systems work that separates a prototype from a production platform. Yojji’s backend teams build exactly this kind of resilience into the microservices they ship for clients across Europe, the US, and the UK.

Yojji is an international custom software development company founded in 2016, specializing in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and scalable microservices architecture. If your next project needs backend performance that holds up under real traffic, they are worth talking to.