The Practical Developer

SLOs Without The Theatre: How To Pick Three Numbers That Actually Help

December 8, 2023 by The Practical Developer

Most teams adopt SLOs by copying Google's book and end up with 30 dashboards nobody reads. The version that earns its keep is two SLIs per service, an error budget that drives real decisions, and a quarterly review. Here is the working setup and the rule that keeps SLOs from becoming bureaucracy.

reliability

sre

process

A laptop screen with charts — the right setting for the kind of measurements an SLO is built on

The team reads the Google SRE book and decides to “do SLOs.” Three months later, every service has 8 SLIs, dashboards everywhere, and nobody has actually used the SLO data to make a decision. The error budget exists in a spreadsheet that gets updated quarterly. Reliability has not measurably improved.

SLOs done well are not bureaucratic. They are two numbers per service and one rule that applies them: when the error budget is gone, stop shipping features and fix reliability instead. That is the entire mechanism. Everything else is optional.

This post is the version that works for normal-sized teams: how to pick the SLI, set the SLO, calculate the error budget, and tie it to actual decisions.

The terms, briefly

SLI (Service Level Indicator). A measurement. “Percentage of HTTP requests that returned 2xx within 500ms.”

SLO (Service Level Objective). The target. “99.5% of requests over the trailing 30 days.”

Error budget. The gap between perfection and the SLO. If your SLO is 99.5%, your error budget is 0.5% — over 30 days, that’s 216 minutes of allowed badness. If you’ve used 200 minutes, you have 16 minutes of error budget left.

SLA (Service Level Agreement). A contract. “We refund customers if we drop below 99%.” Looser than the SLO; SLO is internal target, SLA is what you promise outside.

Most teams confuse SLO and SLA. The SLA exists for sales / legal; the SLO is the internal target. SLO is always tighter than SLA.

Pick two SLIs per service

The biggest mistake is having too many SLIs. The right number is two. Pick one availability SLI and one latency SLI:

Availability SLI: percentage of valid requests that succeeded.

SLI = (valid_responses_2xx_3xx + valid_responses_4xx_excluding_429) / total_valid_responses

The 4xx exclusion is important — a user sending a malformed request is not a service failure. 429 (rate limited) is included because it’s the service deciding to fail; 400 / 401 / 403 / 404 are typically excluded.

Latency SLI: percentage of valid requests served within a threshold.

SLI = (requests_under_500ms) / (total_valid_requests)

500ms is a starting point. Pick a number based on user expectations for your service.

For services that aren’t HTTP (queue consumers, batch jobs), the SLIs are different but the same idea: one “did it work” SLI, one “did it work fast enough” SLI.

Pick the SLO based on what users feel

A common trap: setting the SLO at “the current observed level.” If the service is currently at 99.97% availability, setting an SLO of 99.95% means the team is below SLO before they start.

The right SLO is the level at which users start to notice and complain. For most consumer-facing APIs:

  • 99.9% — about 43 minutes/month of downtime allowed. Standard for non-critical SaaS.
  • 99.95% — 22 minutes/month. For revenue-critical paths.
  • 99.99% — 4 minutes/month. Hard to achieve; requires significant investment.
  • 99.999% — 26 seconds/month. Rare; only for infrastructure.

99.9% is the right SLO for most teams. Higher requires actual engineering investment in resilience; you can’t get there by copying a number.

The error budget rule

Here is the entire mechanism: when you have used your error budget, stop shipping features and fix reliability.

If your SLO is 99.9% and you’ve spent 50 minutes of downtime in the last 28 days (out of 40 minutes allowed), you are over budget. The next sprint is not a new feature; it is the work that prevents the next 50 minutes.

This rule is what makes SLOs valuable. Without it, the SLO is decoration — interesting numbers that don’t change anyone’s behavior. With it, the SLO is a forcing function for engineering investment.

The quarterly review

Once a quarter, review the SLOs:

  • Did we hit our targets? If not, why?
  • What did we burn the budget on? (Outages, deploys, dependency failures.)
  • Should the targets change? (Upward if you’re consistently exceeding, downward if you’re consistently failing because of unfixable factors.)
  • What investment moves the needle next quarter?

The right cadence is quarterly. Monthly is too noisy; annual is too slow.

Implementation: Prometheus + Grafana

A working SLO measurement stack is Prometheus collecting metrics and a small set of dashboards. The availability burn-rate query:

1 - (
  sum(rate(http_requests_total{job="api", status!~"5.."}[28d]))
  /
  sum(rate(http_requests_total{job="api"}[28d]))
)

For burn-rate alerts, Sloth or similar generators turn an SLO definition into Prometheus alert rules:

# sloth definition
service: "api"
slos:
- name: "availability"
  objective: 99.9
  sli:
    events:
      error_query: 'sum(rate(http_requests_total{status=~"5.."}[{{.window}}]))'
      total_query: 'sum(rate(http_requests_total[{{.window}}]))'
  alerting:
    page_alert: { labels: { severity: page } }
    ticket_alert: { labels: { severity: ticket } }

Sloth generates the alert rules: page on a fast burn (using budget too quickly), ticket on a slow burn. The alerts say “you’re going to be out of budget in X hours if this continues,” which is meaningful.

Multi-burn-rate alerts

A naive “alert when SLI is below SLO” fires on every blip. The better pattern is multi-burn-rate: alert when you’re burning budget too fast relative to the time window.

  • High burn rate over 5 min → page (something is wrong right now).
  • Medium burn rate over 1 hour → page (sustained degradation).
  • Low burn rate over 6 hours → ticket (slow leak, fix this week).

The math: if your SLO is 99.9% (0.1% error budget), a burn rate of 14× means you’d consume the entire 30-day budget in ~50 hours. That’s the threshold for a fast-burn page.

Sloth, Pyrra, Nobl9 all generate these alerts from a simple SLO definition.

SLOs for things that aren’t HTTP

For queue consumers:

  • Availability: 99.9% of jobs complete (after retries) within their TTL.
  • Latency: 99% of jobs complete within N seconds of being enqueued.

For batch jobs:

  • Availability: % of scheduled runs that complete successfully.
  • Latency: % of runs that complete within their SLO duration.

For a database:

  • Availability: % of queries that succeed.
  • Latency: % of queries under N ms.

The pattern is the same: pick “did it work” and “fast enough.”

The four traps

1. Tracking too much. 30 SLIs across 10 services, nobody knows which to look at first. Two SLIs per service. Roll up across services for an org-level view.

2. SLO too tight. “We aim for 99.99%” but the team is paged every 3 days. The SLO is wrong; nobody trusts it; it gets ignored. Set the SLO at “what we can credibly hit,” not “what we wish we could.”

3. Ignoring the rule. SLO violated, error budget gone, the team ships a new feature anyway because there’s a deadline. Now the SLO is theatre. The rule is the value; if you can’t enforce it, the SLO is decoration.

4. SLO based on infrastructure, not user impact. “99.9% pod uptime” is not what users care about. Measure user-facing requests, not internal pod health.

When to set SLOs

A common question: how big should the team be before SLOs are useful? Roughly: when there are enough services that prioritization between “feature work” and “reliability work” needs a process. Below that — say 5 engineers — SLOs are overkill.

For a 20-engineer org, SLOs help. For a 100-engineer org, they’re essential. For a 5-engineer startup, just have good monitoring and trust.

The takeaway

Two SLIs per service. SLO at “users notice and complain” level. Error budget determines what the team works on next. Quarterly review. Multi-burn-rate alerts so you know when budget is being spent fast.

That is the whole system. Anything more is process the team will eventually drop. Anything less is dashboards without consequence. The middle is the version that pays back.

A note from Yojji

The kind of reliability process that prevents “we hope it’s reliable” from masking a real problem — SLOs, error budgets, the cadence that ties them to engineering investment — is the kind of long-haul engineering culture Yojji’s teams put into the platforms they ship for clients.

Yojji is an international custom software development company founded in 2016, with teams across Europe, the US, and the UK. They specialize in the JavaScript ecosystem (React, Node.js, TypeScript), cloud platforms (AWS, Azure, GCP), and full-cycle product engineering — including the SRE-adjacent practices that decide whether reliability is measured or just hoped for.