Tools
Zone-loss calculator
Capacity headroom after you lose the largest failure domain, and what that spare CPU costs.
Spread a service across failure domains and it still has to live through losing the biggest one. This page takes the nodes you have and the CPU you run them at, and shows what the survivors are doing after that domain is gone. A zone here is any failure domain you can lose as a unit: an availability zone, a rack, a cell. It runs in your browser. Nothing you type is sent anywhere.
Headroom is spare capacity you pay for so that loss doesn't spend theerror budgetall at once. The formulas are below the calculator.
The math
Losing a failure domain doesn't shrink the traffic. It shrinks the number of nodes that can serve it, so the same work sits on fewer machines. Each node here is one unit of capacity, and the nodes are the same size.
surviving nodes = total nodes − nodes in the largest zone
survivor CPU = steady-state CPU × surge × total nodes / surviving nodesWhen the zones are even and you aren't counting a retry surge, the zone sizes cancel. N is the number of zones.
survivor CPU = steady-state CPU × N / (N − 1)Three even zones at 60% land at 60% × 3/2 = 90%. Three even zones at 40% land at 40% × 3/2 = 60%. The first is a bad moment for a retry storm. The second still looks like a normal day.
Zones that aren't even use the same formula with the real counts. A 2/2/1 layout has 5 nodes and loses 2, so the multiplier is 5/3. At 60% the survivors are at 100%. At 40% they're at 66.67%, which is still under an 80% line.
Leave the surge at 1 and the requests simply move. Set it to 1.5 if the failure makes clients retry enough to add half again as much work. The surge applies to the loss, not to the steady state you run on a quiet day. A layout that looks fine at 1× can be over the line once retries show up.
The highest target that still survives
The danger line is the utilization you don't want the survivors to cross. The highest steady-state target that lands on that line is the line, scaled by how much capacity the loss leaves you.
max safe CPU = danger line × surviving nodes / (total nodes × surge)Even zones, surge of 1:
max safe CPU = danger line × (N − 1) / NAn 80% line and 3 zones gives 80% × 2/3, which is 53.33% after the calculator rounds it down. The printed target stays on the safe side of the line. Run hotter than that and a single zone loss crosses the line you set.
What the extra nodes cost
Offered load stays put. You're buying nodes so today's traffic is a smaller share of a larger fleet. You need enough survivors that the loss, including the surge, lands on the danger line or under it.
steady-state CPU × current nodes × surge / new surviving nodes ≤ danger line
monthly cost of the headroom = extra nodes × cost per node per monthThe count is the smallest number of nodes that gets you there. New nodes go on the smaller zones. A node you add only to the largest zone is in the domain you just assumed you'd lose, so it doesn't move the surviving count. When the zones are already even, one extra node creates a new largest zone and the survivors don't gain capacity. The next node, on a different zone, does. You can't buy a fraction of a node, so the fleet usually lands a little under the maximum safe target, not on it.
Headroom is the cheapest error budget you'll be offered. You're paying for spare CPU so one failure domain doesn't turn into user-visible errors. Efficiency, the CPU a request costs, is the dial that lets you buy that headroom: a cheaper request means the same spare capacity covers more traffic, or the same traffic needs a smaller bill. Spend it on headroom when the service is eating its error budget. Take the cash back only when latency and the budget are both comfortable.
One zone can't survive its own loss. The surviving count is zero, and more nodes in that same zone stay in the zone you lose. N-1 starts at two failure domains.
This is a model, not a forecast. It assumes the leftover traffic spreads across the nodes that remain, and that nothing autoscales during the failure. CPU is the label because that's the usual resource, but the arithmetic is the same for anything you scale by adding equal nodes. It doesn't replace an SLO. It tells you whether a zone loss will shove the service into the part of the curve where latency and errors spend the budget. The error budget calculator is that other half. The longer writeup of demand, headroom, and lead time will live in capacity planning.
What's in the link
Copy link puts the current state in the query string. nodes is a slash-separated list when the zones aren't even, such as 2/2/1. Commas work too. When they are even, zones is the count and per is the nodes in each one. cpu is the steady-state percent, danger is the danger line, surge is the retry multiplier, and cost is dollars per node per month. A field that's omitted is the default: 3 zones, 2 nodes each, 60% CPU, an 80% danger line, a surge of 1, and no price.
Further reading: Sean Wragg, Efficiency is an error-budget input.