Skip to content

Guide

An SLO and error budget primer

How to define a service level indicator, set a target below 100%, and use the error budget that target creates.

SRE Academy8 min readSLOs and error budgets

An SLO is a choice about how unreliable a service is allowed to be. The error budget is what’s left after you make that choice. This guide is for a team that has heard the words and still doesn’t have a written target.

Three terms that get swapped

An SLI, a service level indicator, is a measurement of the service a person actually receives. “The proportion of valid checkout requests that succeed within 300 milliseconds, measured at the edge” is an SLI once every word in that sentence is defined. A graph of CPU is not an SLI. Users don’t experience CPU. They experience a page that loads, a charge that completes, or a request that fails.

An SLO, a service level objective, puts a target on that measurement over a window you name. “99.9% of those requests succeed inside that latency bound, over a rolling 28-day window” is an SLO. It’s an internal target. The people who build the service, the people who operate it, and the product owner should be able to point at the same sentence.

An SLA, a service level agreement, is a promise to a customer that includes a consequence if you miss it: a credit, a penalty, a contract conversation. You don’t need an SLA to have an SLO. If you do have an SLA, the internal SLO should be tighter, so you notice trouble before the contract does. Calling an internal target an SLA doesn’t make it one.

Write the indicator as a ratio

A usable SLI is a ratio of good events to valid events.

Good events met the bar. A successful response inside the latency threshold. A job that finished before its freshness limit. A read that returned the bytes it was supposed to return.

Valid events are the denominator: the events you meant to serve.

A ratio survives a gap in the data better than a success counter does. If the pipeline that records the ratio stops, the ratio stops moving, and the gap is visible. A counter that only increments on success will sit still and look calm while you’re blind.

Be explicit about what you leave out of the denominator. It’s reasonable to exclude traffic you never intended to serve, such as malformed requests, health checks, or a client that canceled before you could answer. Write the exclusion down. It isn’t reasonable to exclude your own failures by defining them as invalid. A 500 that you returned is a valid event that wasn’t good.

Measure as close to the user as you can defend. The edge, or the client, is what people actually hit. A success rate on one internal hop can be green while the hop in front of it is failing. If you only have a server-side measurement, say that in the SLI. An honest proxy is better than a vague one.

For latency, use a threshold and a percentile, not a mean. A mean of 80 milliseconds can hide a tail of requests that take seconds, and users live in the tail. “99% of valid requests complete in under 400 milliseconds” is a sentence you can test. The number 400 is an illustration, not a recommendation. A batch billing job and a keystroke in an editor don’t share a threshold. Pick the point where a user, or the system that depends on you, would call the response late.

The target is how much failure you’re buying

100% is the wrong target for almost every user-facing system. The last fraction of a percent costs more than the ones before it, in engineering time and in changes you’re then afraid to ship. It also leaves no room for the work a reliability target is supposed to allow: deploys, migrations, and experiments.

An SLO of 99.9% is a decision that 0.1% of the window may be bad. That 0.1% is the error budget. You want the budget large enough to operate and small enough that users, and the business, can live with the misses. There is no universal percentage. A payments path and an internal dashboard shouldn’t wear the same number just because 99.9 is familiar.

Two windows show up often, for a practical reason. A rolling 28 days or a rolling 30 days smooths a weekly pattern. A calendar month jumps around, because February and March aren’t the same length, and a bad day at the boundary falls out of the window all at once. Pick one window and use it whenever you talk about the budget. If you change the window, say that you changed it.

Don’t set the target by copying last quarter’s graph and rounding. That number is a baseline. It’s useful input. It isn’t automatically the right promise. Last quarter may have been quiet, or it may have been bad.

Minutes and requests are different budgets

People talk about error budgets in minutes. That’s the right unit only when the indicator is time-based availability: the service is down for everyone, or it isn’t. A lot of user-facing failure is partial. Some requests fail and the rest don’t. For a request-based SLI, the budget is a share of requests, not a stretch of clock time.

The error budget calculator does this arithmetic for any target and window, and it turns a request rate into a bad-request budget.

Write the policy before you need it

The error budget is a pile of allowed failure. Spending some of it on a launch isn’t a moral failure. It’s the reason you didn’t choose 100%. The policy is the agreement about what happens as the pile shrinks, written when nobody is in an incident.

Answer these before you need the answers:

  1. Who may spend the budget on purpose? Launches, experiments, and maintenance all consume it. Name the path those changes take.
  2. What happens when the burn is fast, even if the budget isn’t gone? The usual answer is to stop the risky change, mitigate, and look at what consumed the budget.
  3. What happens when the budget for the window is exhausted? A common default is that non-essential releases wait until the rolling window recovers. Whatever your answer is, it should be the default, not a debate you reopen at midnight.
  4. Who can grant an exception, and where is it recorded? An exception that lives only in a chat scroll is how the policy quietly ends.

This is a product decision as much as an engineering one. If only the on-call team knows the policy exists, it isn’t a policy yet.

Alert on the burn, not on the host

A single saturated machine is a symptom. Paging a person for every symptom is how a rotation learns to ignore pages. Page when the error budget will be gone soon if the current rate continues.

That rate has a name. A burn rate of 1, held constant, exhausts the budget exactly at the end of the window. A burn rate of 10 exhausts it in a tenth of the window. Fast burns should page. Slow burns should become a ticket while there’s still budget left to save.

The SRE workbook’s chapter on alerting describes multi-window, multi-burn-rate alerts, and it’s a solid starting point. A fast burn has to show up over a short window and a slightly longer one before you page, so one noisy minute doesn’t wake anyone. A slow burn opens a ticket. The workbook’s specific windows are a published starting point, not a law. Change them when you know your traffic and your paging load. Keep the idea: page on how fast the budget is disappearing, and require the burn to be visible at two timescales. The error budget calculator scales those burn rates to your window and drafts the Prometheus rules.

If you don’t have a burn alert yet, you can still operate the SLO. Look at the remaining budget on a cadence a person will actually keep. An SLO that only appears in a quarterly slide hasn’t been adopted.

Keep the set small, and review it

One user journey usually needs a short list. Availability of the critical request, and its latency, covers a lot of products. Forty SLOs is a monitoring program with a new name. You won’t make a release decision against forty numbers.

Put a review date in the document. The review is a few direct questions:

  • Did we meet the target? If you can’t tell, the indicator isn’t measured yet. Fix that before you edit the number.
  • Did users complain while the SLO was green, or stay quiet while it was red? The indicator isn’t the thing they feel.
  • Did the policy change a decision this window? If it never does, the budget is decorative, or the team is afraid to spend it.
  • Did anyone repair a miss by changing the definition after the fact? Treat that as its own problem and write it down.

What to do this week

  1. Pick one user journey, not the whole company.
  2. Write the SLI as good events over valid events. Name where it’s measured and what the denominator excludes.
  3. Set a target below 100%, and a rolling window. Compute the budget in the unit the SLI actually uses.
  4. Write the four policy answers. Have the person who can stop a launch read them.
  5. Put the remaining budget somewhere the team looks during the week. Add a burn alert when you can.
  6. Copy the SLO document template so the definition doesn’t live in a slide.

When the budget is gone and the policy says to wait, waiting is the practice. The primer isn’t the hard part.

Further reading

The public sources are worth reading in full. This guide is a starting map, not a replacement.

Written by SRE Academy. Published .

Templates for this topic

  • SLO document template

    A blank SLO specification. It asks for the indicator, the target, the window, and what happens when the budget is gone.

More guides