Load Shedding in Node.js: How to Reject Traffic Before You Drown

Your Redis cluster hiccups. Latency jumps from 2ms to 200ms. Requests that used to take 80ms now take 800ms. Because nothing in your app says “no,” the queue of in-flight work grows. Memory climbs. The event loop lag rises. Health checks start failing. Kubernetes restarts the pods. The restart floods the already-slow Redis with reconnections. Now every pod is down.

That is not a Redis problem. It is a load shedding problem.

Rate limiting protects you from malicious clients. Circuit breakers protect you from broken dependencies. Bulkheads protect one route from another. Load shedding protects you from yourself: the tendency to accept every request and queue it until the process collapses. This post is the admission controller you need in Node.js, the metrics that tell you when to activate it, and the load-test proof that rejecting 10% of traffic early keeps the other 90% healthy.

The queue death spiral

Node.js looks resilient because it does not block on I/O. But it absolutely queues I/O. Every request that arrives while another is waiting for Redis, Postgres, or an HTTP call sits in memory. The event loop is still ticking, but the promises are piling up.

The spiral looks like this:

1. Dependency latency rises (cold cache, network blip, database vacuum).
2. In-flight requests accumulate because new work keeps arriving.
3. Memory grows. GC pressure rises. Event loop lag increases.
4. Timeouts start firing. Retries fire more requests into the same slow dependency.
5. Health checks, which hit the same dependency, begin timing out.
6. The orchestrator kills “unhealthy” pods.
7. Remaining pods absorb the traffic and die faster.

The fix is not to scale up. If you add pods while the dependency is slow, you amplify the pressure against it. The fix is to shed load: reject or defer requests before they enter the system, so the requests you do accept finish quickly and keep the service alive.

Admission control: the core idea

Admission control is a bouncer at the door of your service. It looks at two things before letting a request in:

1. Is the service currently healthy enough to handle this work?
2. Is there capacity to process this request within a reasonable deadline?

If the answer to either is no, the request gets a fast 503 with a Retry-After header. The client retries with backoff. The service stays stable. The user sees a brief delay, not a 30-second timeout.

This is different from rate limiting. Rate limiting cares about the client. Admission control cares about the server.

What to measure

You cannot make a load shedding decision without real-time signals. The minimal set is three metrics:

• Event loop lag (eventLoopUtilization or eventLoopDelay). Tells you if the process itself is choked.
• In-flight request count (gauge). Tells you how much work is already inside.
• Dependency latency (rolling p95). Tells you whether the current traffic is finishable.

Optional but useful:

• Memory usage as a percentage of the container limit.
• CPU utilization (on systems where CPU is throttled, not the event loop).

These must be cheap to read. You cannot afford a heavy metrics query on every request.

The admission controller middleware

Here is a practical admission controller for Express or Fastify. It uses event loop lag and in-flight count to make a fast binary decision: admit or shed.

import { eventLoopUtilization } from 'node:perf_hooks';
import { setTimeout } from 'node:timers/promises';

class AdmissionController {
  constructor(options) {
    this.maxInflight = options.maxInflight ?? 100;
    this.targetLagMs = options.targetLagMs ?? 50;
    this.lagHistoryMs = options.lagHistoryMs ?? 500;
    this.shedProbability = options.shedProbability ?? 1.0;
    this.lagThreshold = options.lagThreshold ?? 100;

    this.inflight = 0;
    this.elu = eventLoopUtilization();
  }

  getLagMs() {
    const next = eventLoopUtilization(this.elu);
    this.elu = next;
    // utilization is 0..1. Convert to rough lag estimate
    // More precise: use histogram-based eventLoopDelay, but this is fast.
    return Math.round((next.utilization * this.lagHistoryMs));
  }

  shouldShed() {
    const lag = this.getLagMs();
    const full = this.inflight >= this.maxInflight;
    const lagging = lag >= this.lagThreshold;

    if (full) return true;
    if (lagging) {
      // Probabilistic shedding spreads retry storms across clients
      return Math.random() < this.shedProbability;
    }
    return false;
  }

  async middleware(req, res, next) {
    if (this.shouldShed()) {
      res.setHeader('Retry-After', '2');
      res.status(503).json({ error: 'Server overloaded, retry soon' });
      return;
    }

    this.inflight++;
    const cleanup = () => this.inflight--;
    res.on('finish', cleanup);
    res.on('error', cleanup);

    next();
  }
}

Use it globally:

const admission = new AdmissionController({
  maxInflight: 80,
  targetLagMs: 50,
  lagThreshold: 80,
  shedProbability: 0.2, // shed 20% when lagging
});

app.use((req, res, next) => admission.middleware(req, res, next));

Why probabilistic shedding? If you shed 100% of traffic the instant lag crosses 80ms, you create a cliff: clients retry together, hit the exact same threshold, and retry again. Probabilistic shedding smooths the recovery curve. You can tune shedProbability from 0.1 up to 1.0 as lag worsens.

Making it adaptive with a PID-style controller

A fixed threshold works, but traffic patterns change. A better approach is an adaptive controller that adjusts maxInflight based on real latency.

class AdaptiveAdmissionController extends AdmissionController {
  constructor(options) {
    super(options);
    this.currentMax = options.maxInflight ?? 100;
    this.minMax = options.minMax ?? 20;
    this.maxMax = options.maxMax ?? 200;
    this.adjustIntervalMs = options.adjustIntervalMs ?? 5000;
    this.p95LatencyMs = 0;
    this.targetLatencyMs = options.targetLatencyMs ?? 200;

    setInterval(() => this.adjust(), this.adjustIntervalMs).unref();
  }

  recordLatency(ms) {
    // Simple exponential moving average for p95 approximation
    this.p95LatencyMs = Math.max(ms, this.p95LatencyMs * 0.7 + ms * 0.3);
  }

  adjust() {
    const error = this.p95LatencyMs - this.targetLatencyMs;
    const step = Math.round(error / 10); // proportional control

    this.currentMax = Math.max(
      this.minMax,
      Math.min(this.maxMax, this.currentMax - step)
    );

    // Reset p95 estimate after adjustment to avoid over-correction
    this.p95LatencyMs *= 0.5;
  }

  shouldShed() {
    return this.inflight >= this.currentMax || this.getLagMs() >= this.lagThreshold;
  }
}

This controller lowers capacity when latency rises and raises it when latency falls. The currentMax value is the only state you need to export to metrics. A Grafana alert on currentMax < 30 tells you the service is under sustained pressure and needs investigation, not just more pods.

Per-route shedding vs global shedding

Global shedding is the minimum. But some routes are more important than others.

• Critical routes (/checkout, /login, /health): higher maxInflight, lower shedProbability. You want these to survive even when analytics and exports are dying.
• Degradable routes (/export, /analytics, /search): lower limits, higher willingness to shed. These are allowed to queue or fail.

Use the bulkhead pattern from an earlier post: separate admission controllers per route group.

const criticalAdmission = new AdmissionController({ maxInflight: 120, shedProbability: 0.1 });
const backgroundAdmission = new AdmissionController({ maxInflight: 20, shedProbability: 0.8 });

app.get('/checkout', (req, res, next) => criticalAdmission.middleware(req, res, next), checkoutHandler);
app.get('/export', (req, res, next) => backgroundAdmission.middleware(req, res, next), exportHandler);

When the database slows down, exports get rejected fast while checkout keeps processing. That is the difference between a partial outage and a total outage.

What the client should see

A shed request must be unambiguous. Never return 200 with a delayed timeout. Return 503 with a Retry-After header.

{
  "error": "Server temporarily overloaded",
  "retry_after": 2
}

Client libraries and browsers respect Retry-After. Mobile apps should implement exponential backoff with jitter. The server is not lying; it is admitting the truth that it cannot accept this work right now.

How to test it

Load shedding is not a unit test. It is a load test. You need to prove that the service stays healthy when dependency latency doubles.

Use autocannon or k6 with two phases:

1. Baseline: 50 RPS against a fast dependency. All 200s, p99 under 100ms.
2. Degraded: same 50 RPS, but the dependency now takes 400ms (inject a sleep).

Without admission control, you see:

• p99 climbs to multiple seconds
• Memory grows linearly
• Eventually timeouts and 500s

With admission control:

• A percentage of requests returns 503 in under 1ms
• Admitted requests still finish in under 500ms
• Memory stays flat

That is the graph you show in the incident review: “Yes, traffic was bad. Yes, we rejected 15%. But the service never fell over.”

Integration with your orchestrator

Load shedding protects your process. Your orchestrator must not fight it.

• Health checks should hit a dedicated /health endpoint that bypasses admission control, or use a separate, very lenient controller. If Kubernetes kills pods because /health was rejected, you amplify the problem.
• Startup probes must succeed before the main admission controller activates. A warming pod should not shed traffic.
• HPA should scale on CPU or custom metrics, not request count. If you scale on request count during a shedding event, you add pods that also start shedding because the dependency is still slow.

The ideal metric for autoscaling is CPU + event loop lag. If both are high, you need more capacity. If event loop lag is high but CPU is low, the dependency is slow and scaling will not help.

Monitoring the shedder

You need four metrics in your dashboard:

• admission_rejected_total (counter, tagged by route and reason)
• admission_inflight (gauge)
• admission_max_inflight (gauge, shows adaptive controller state)
• event_loop_lag_ms (gauge)

Alert on trends:

• rejected_total increasing for 5+ minutes: investigate the dependency.
• inflight pinned at max_inflight + p95 rising: the controller is working but you are near collapse.
• event_loop_lag > 100ms with low CPU: a dependency is wedged.

Practical defaults you can copy

• maxInflight: max(2 * vCPU cores * 20, 80) for API servers
• lagThreshold: 50ms for latency-sensitive APIs, 100ms for background workers
• shedProbability: 0.1 at threshold, 0.5 at 2x threshold, 1.0 at 3x
• Retry-After: 2 seconds (clients should add jitter)
• Re-evaluate every 5 seconds if adaptive

Tune these with real load tests. Do not copy numbers from blog posts into production without proving them under your traffic shape.

The closing argument

Every distributed system fails in the same way: it accepts more work than it can finish, queues it, and dies trying. Load shedding is the admission of that reality. It says, “We are full. Come back in two seconds.” That rejection is a better user experience than a 30-second timeout, a better operational outcome than a cascading restart loop, and a better engineering decision than hoping the database gets faster.

Build the controller. Run the load test. Set the alerts. The next time your cache tier blips, your service will blink, not break.

---

A note from Yojji

Engineering teams that ship resilient backend systems treat load shedding as a baseline control, not a crisis-only switch. The same kind of operational rigor, from admission control to adaptive capacity tuning, is what Yojji brings to backend and cloud-native builds where traffic spikes are a matter of when, not if.

Yojji is an international custom software development company founded in 2016, with offices in Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and the reliability patterns that keep production systems running when dependencies misbehave.