The Practical Developer

Kubernetes Liveness And Readiness Probes: The Difference That Causes Half Your Outages

January 20, 2023 by The Practical Developer

Most teams configure liveness and readiness probes identically and wonder why a slow database makes Kubernetes restart their pods in a death spiral. Here is what each probe is actually for, the right endpoint shape for each, and the four-line config that turns an outage into a non-event.

kubernetes

devops

reliability

A laptop full of dashboards and metrics — the moment a Kubernetes restart loop becomes visible

Postgres has a brief blip. Connections fail for 30 seconds. The application’s health endpoint, which queries the database, starts returning 500. Kubernetes’ liveness probe sees the 500 and restarts every pod. Now you have zero healthy pods, every pod is restarting at once, and the brief blip has become a 10-minute outage.

This is the classic liveness-probe death spiral, and it happens because most teams treat liveness and readiness as the same thing. They are not. They have different responsibilities, different failure semantics, and need different endpoint shapes. Getting the difference right is one of those Kubernetes details that turns an outage into a non-event.

What each probe means

Liveness probe. “Is this process alive, or is it stuck and needs a kill?” A failed liveness probe causes Kubernetes to restart the pod. The right reason for liveness to fail is something a restart will fix: a deadlock in the application, a stuck event loop, a process that has wedged itself.

Readiness probe. “Is this pod ready to receive traffic?” A failed readiness probe causes Kubernetes to remove the pod from the service’s load balancer — but the pod keeps running. The right reasons for readiness to fail are transient: starting up, dependency unavailable, paused for graceful shutdown.

Startup probe. “Is this pod still booting?” Used in front of liveness so that a slow-starting pod does not get killed before it finishes initializing. Configurable per-app.

The death spiral happens when teams point liveness at an endpoint that checks the database. Database has a blip → liveness fails → pod restarts → restart does not fix the database blip → next pod’s liveness fails → cascade.

The right endpoint shapes

Two endpoints, two responsibilities.

/health/live — liveness:

app.get('/health/live', (req, res) => {
  // Only check things a restart can fix.
  // Not: "can I reach the database?"
  // Yes: "is the event loop responsive?"
  res.status(200).send('ok');
});

For most apps, returning 200 ok is enough. The probe answers “is the process responsive?” If the HTTP server can respond, the answer is yes. If you have a worker that can deadlock, add an in-process heartbeat counter and check it has incremented recently.

/health/ready — readiness:

app.get('/health/ready', async (req, res) => {
  try {
    // Check that downstream dependencies the pod cannot serve traffic without are reachable.
    await db.query('SELECT 1', { timeout: 2000 });
    await redis.ping();

    if (state.shuttingDown) {
      return res.status(503).send('shutting down');
    }

    res.status(200).send('ready');
  } catch (err) {
    res.status(503).send('not ready');
  }
});

Readiness can fail freely. Failing readiness pulls the pod out of the load balancer; it does not kill it. When the database recovers, the next probe succeeds, the pod re-joins the LB. No restart.

A correct config

spec:
  containers:
  - name: api
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 1     # readiness flaps quickly
      successThreshold: 1
    startupProbe:
      httpGet:
        path: /health/live
        port: 8080
      periodSeconds: 5
      failureThreshold: 60    # 60 × 5s = 5min to start

A few non-obvious choices:

  • failureThreshold: 3 on liveness. Three consecutive failures, 10s apart = 30 seconds of grace. A transient hiccup does not kill the pod.
  • failureThreshold: 1 on readiness. One failure pulls the pod out of the LB. Readiness should be jumpy — that is the point.
  • Separate startupProbe. Lets initialDelay be small for liveness and lets a slow-starting app have a long grace before liveness starts checking.
  • timeoutSeconds: 2. If your liveness endpoint takes >2s to respond, the probe fails. Keep it cheap.

What “graceful shutdown” needs from probes

When Kubernetes terminates a pod (rolling deploy, scale-in, evicted), it sends SIGTERM. Your app should:

  1. Mark itself not-ready (set state.shuttingDown = true).
  2. Wait ~10 seconds for in-flight requests and for the load balancer to remove the pod (readiness probe runs every 5s).
  3. Drain active connections.
  4. Exit.

The terminationGracePeriodSeconds: 30 field gives you 30s before SIGKILL. The preStop hook can run a sleep:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 10"]

This buys time for the readiness probe to mark the pod down and the LB to stop sending traffic before your app starts dropping connections. Without it, you get 502s during every deploy. (See the graceful-shutdown post for the Node.js side.)

What to NOT put in liveness

A non-exhaustive list of things teams put in liveness that they shouldn’t:

  • Database query. Database blip → cascade restart.
  • Redis ping. Redis blip → cascade restart.
  • Downstream HTTP call. External outage → your service restarts itself, doesn’t help.
  • Cache warm-up check. Restart wipes the cache. Use readiness instead.
  • Disk space check. Restart does not free disk. Set up a separate alert.
  • Memory threshold. Restart frees the memory but masks the leak. Find the root cause.

The single rule: if the failure cause is external, do not check it from liveness. External failures should fail readiness, not liveness.

The tricky case: actual deadlocks

If your application can wedge itself — a deadlocked promise, a stuck synchronous loop, a worker that has stopped processing — liveness needs to detect it. The pattern: an in-process heartbeat that the main loop updates, and a liveness handler that checks it.

let lastHeartbeat = Date.now();

// Main loop / event handler increments the heartbeat.
setInterval(() => { lastHeartbeat = Date.now(); }, 1000);

app.get('/health/live', (req, res) => {
  if (Date.now() - lastHeartbeat > 30_000) {
    return res.status(500).send('event loop stalled');
  }
  res.status(200).send('ok');
});

If the event loop is responsive, the interval runs and lastHeartbeat is recent. If it has stalled (CPU-bound code, deadlock), lastHeartbeat ages and liveness fails. Restart fixes the wedge.

For worker processes that don’t have an HTTP server, write a heartbeat file and have a sidecar HTTP server check its mtime. Same shape.

What Kubernetes events tell you

When liveness keeps failing, kubectl describe pod shows it:

Warning  Unhealthy  ... Liveness probe failed: HTTP probe failed with statuscode: 500
Normal   Killing    ... Container api failed liveness probe, will be restarted

If you see a series of these, two questions:

  1. Is liveness checking something it shouldn’t? (database, redis, external API)
  2. Is the timeout too tight? (timeoutSeconds: 1 will fail a slow-but-alive pod)

Most “we are restart-looping” incidents are one of those two.

Probe behavior across rolling deploys

A correctly configured probe stack makes rolling deploys boring:

  1. New pod starts. Startup probe runs. Eventually succeeds.
  2. Liveness takes over. Steady-state.
  3. Readiness checks dependencies. When ready, pod joins the LB.
  4. Old pod gets SIGTERM. preStop sleeps. Readiness drops to 503. LB drains.
  5. preStop completes. App shuts down. Pod removed.

If any of these stages is wrong — probe shape, timing, graceful shutdown — you get 502s, dropped connections, or restart loops. The whole system has to fit together.

The takeaway

Kubernetes’ two-probe model is more sophisticated than most teams use. Liveness restarts; readiness drains. Liveness should check only what a restart fixes; readiness can check everything that affects whether traffic should be routed here. The death-spiral pattern of pointing liveness at the database is one of the most common production misconfigs, and getting it right is a four-line change to the manifest.

Spend an afternoon refactoring your /health endpoints into /health/live and /health/ready. Set failureThreshold correctly for each. Test what happens when the database is paused. The next dependency outage you have will be a 30-second non-event instead of a 10-minute outage.

A note from Yojji

The kind of platform-engineering detail that turns a downstream blip into a non-event — probe shape, graceful shutdown, restart semantics — is the unglamorous DevOps work that decides whether a deploy is uneventful or front-page. It is the kind of work Yojji’s teams build into the Kubernetes platforms they ship for clients.

Yojji is an international custom software development company founded in 2016, with offices across Europe, the US, and the UK. Their teams specialize in the JavaScript ecosystem, cloud platforms (AWS, Azure, GCP), and Kubernetes-based deployments — including the probe and rollout configuration that decides whether your service stays up during a database hiccup.