System Atlas
the atlas

System performance metrics every engineer should know

Averages lie; percentiles and the golden signals tell you the truth.

the gist

Performance boils down to a handful of numbers: latency (measured at percentiles, not averages), throughput, error rate, saturation, and availability, with Apdex rolling latency into a user-satisfaction score. The single most important idea is the difference between the average and the tail — your worst requests, captured by p95 and p99, are what actually shape user experience.

ANALOGY // A coffee shop: latency is how long your order takes, throughput is drinks served per hour, error rate is orders made wrong, saturation is how packed the queue is — and averaging out the one customer who waited twenty minutes hides exactly the person who'll never come back.
living diagram01 / 07
The core signals of system performance, in the order you should reach for them
01
Latency
time to serve one request
02
Percentiles / tail
p50 · p95 · p99 — not the average
03
Throughput
requests per second — capacity
04
Error rate
% of requests that fail
05
Saturation
how full a resource is — headroom
06
Availability
uptime in nines
07
Apdex
0–1 user-satisfaction score
01

Latency

Latency is the time to serve a single request — the delay a user feels between asking and receiving. It's the most direct measure of responsiveness. But a single latency number is meaningless without knowing how it's distributed, which is why raw averages are a trap.

02

Percentiles and the tail

Report latency as percentiles: p50 (median) is a typical request, while p95 and p99 are the slow tail — the value 95% or 99% of requests come in under. The tail is what actually shapes experience, because at scale that 'rare' slow 1% happens to millions of users and often to your most active ones. Chasing the average hides your worst behaviour; watching p99 exposes it.

03

Throughput

Throughput is how much work the system does per unit time, usually requests per second (RPS) or queries per second. It measures capacity rather than speed — a system can be fast per request yet low-throughput, or high-throughput with mediocre latency. Together, latency and throughput describe how a system behaves as load rises.

04

Error rate

Error rate is the fraction of requests that fail — HTTP 5xx responses, timeouts, exceptions — usually expressed as a percentage of total traffic. It's the most direct signal of correctness and health; a rising error rate means users are hitting failures right now. It pairs naturally with latency, since a slow system often starts shedding errors under stress.

05

Saturation & utilisation

Saturation measures how full a resource is — CPU, memory, disk I/O, connection pools, queue depth — as a fraction of its maximum. It's the leading indicator of trouble: latency and errors usually stay flat until a resource nears 100%, then degrade sharply. Watching saturation tells you how much headroom remains before performance falls off a cliff.

06

Availability

Availability is the proportion of time the system is up and serving, expressed in 'nines'. 99.9% ('three nines') allows about 8.7 hours of downtime a year; 99.99% allows about 52 minutes; 99.999% about 5 minutes. Each extra nine is roughly ten times harder and costlier to achieve, so targets are chosen deliberately against what the business actually needs.

07

Apdex

Apdex (Application Performance Index) compresses latency into a single 0–1 satisfaction score against a target threshold T. Requests under T count as satisfied, those between T and 4T as tolerating, and slower ones as frustrated. It's a convenient executive-friendly summary, though like any single number it should sit alongside the percentiles, not replace them.

the full picture
01

Why you need metrics at all

"Is the system fast?" is not an answerable question until you make it measurable. Performance work lives or dies on metrics — precise, quantitative signals that tell you how a system is actually behaving rather than how it feels. Without them you're guessing: you can't tell whether a change helped, whether you're near a cliff, or whether the outage you're firefighting is latency, errors, or capacity. With them, you can set targets, alert before users notice, and reason about trade-offs.

The good news is that the space is small. A handful of numbers — latency, throughput, error rate, saturation, and availability — capture the vast majority of what you need, and Google's "four golden signals" (latency, traffic, errors, saturation) and the related RED and USE methods are just curated subsets of them. The single most important skill isn't memorising the list, though; it's learning to *interpret* these numbers correctly, and the biggest interpretation mistake, by far, is trusting averages.

02

Latency, and why the average lies

Latency is the time to serve a single request — the delay between a user asking and receiving. It's the most direct measure of responsiveness, and the instinct is to summarise it with an average: "our average response time is 100 ms." That instinct is dangerous.

The problem is that latency distributions are almost never symmetric; they have a long tail of slow requests. Imagine 99 requests at 50 ms and one at 5000 ms — the average is a reassuring 99 ms, completely hiding that one user waited five seconds. Averages smear your worst behaviour into your best and make it vanish.

The fix is percentiles. The p50 (median) is a genuinely typical request. The p95 and p99 describe the tail: p99 = 500 ms means 99% of requests were faster than 500 ms and the slowest 1% were worse. Percentiles can't be averaged away because they report actual experienced values at the edges of the distribution. Whenever you see a latency figure with no percentile attached, treat it with suspicion.

03

The tail is the experience

Why obsess over the slowest 1%? Because at scale, "rare" events are constant, and they concentrate on the people who matter most. If a single page load makes 100 backend calls, the chance that *at least one* of them hits your p99 latency is enormous — so your p99 per-call latency becomes something close to your *typical* page latency. Slow tails compound.

Worse, tail latency correlates with load. Your heaviest users — the ones with the fullest carts, the most data, the most engagement — tend to trigger the slowest requests, so the tail disproportionately punishes your best customers. This is why large engineering organisations set service-level objectives on p99 (and even p99.9), not on the mean. "Tail latency" is not a niche concern for perfectionists; it is the metric that most directly maps to whether the product feels fast. The discipline is simple to state and hard to live by: optimise and alert on the tail, because that's where your users actually are.

04

Throughput, errors and saturation

Latency describes *how fast*; three more signals describe *how much*, *how healthy*, and *how full*.

  • Throughput is the amount of work done per unit time — requests per second (RPS), queries per second, transactions per second. It measures *capacity*, which is distinct from speed: a system can serve each request quickly yet handle few in parallel, or handle a torrent at mediocre per-request latency. Latency and throughput together describe how a system behaves as load climbs, and they usually trade off — push throughput toward the limit and latency rises.
  • Error rate is the fraction of requests that fail — 5xx responses, timeouts, exceptions — as a percentage of total traffic. It's the most direct correctness signal: a spike means users are hitting failures *now*.
  • Saturation (utilisation) is how full a constrained resource is — CPU, memory, disk I/O, connection pools, queue depth — relative to its maximum. It's the crucial *leading* indicator: latency and errors often stay flat and healthy right up until a resource approaches 100%, and then degrade sharply. Watching saturation tells you how much headroom you have before that cliff, so you can scale *before* the numbers that users feel go bad.
05

Availability, Apdex and using them together

Availability is the proportion of time a system is up and correctly serving requests, conventionally expressed in nines. The scale is punishing: 99% ("two nines") permits ~3.65 days of downtime a year, 99.9% ~8.7 hours, 99.99% ~52 minutes, and 99.999% ("five nines") only ~5 minutes. Each additional nine is roughly ten times harder and more expensive to achieve — requiring redundancy, failover, and relentless operational rigour — so availability targets are a deliberate business decision, not a badge to maximise blindly. A five-nines target for a photo-sharing hobby app would be a waste; for a payments processor it's table stakes.

Apdex (Application Performance Index) rolls latency into a single 0–1 user-satisfaction score against a target threshold T: requests faster than T are "satisfied," those between T and 4T are "tolerating," and slower ones are "frustrated," combined into one number. It's handy for dashboards and non-engineers, but like any single-number summary it can hide detail, so it should complement the percentiles rather than replace them.

The practical way to use all of this is layered. Start with the golden signals — latency (at percentiles), traffic/throughput, errors, and saturation — as your always-on health view. Set SLOs on the tail latency and error rate that reflect real user expectations, alert on saturation to get ahead of trouble, and track availability against a target justified by the business. Above all, keep asking of every number: is this the average smoothing over a painful tail? The engineers who reason well about performance are the ones who never stop distinguishing the typical case from the worst case.

what to remember
01

Latency is how long one request takes; always report percentiles (p50, p95, p99), never just the average.

02

The tail (p99) matters most: at scale, a slow 1% of requests hits users constantly and drags experience down.

03

Throughput (requests per second) measures capacity — how much work the system handles per unit time.

04

Availability is uptime measured in 'nines': 99.9% is ~8.7h/yr down, 99.99% is ~52min/yr.

05

Error rate (% of failed requests) and saturation (how full a resource is) reveal health and headroom.

06

Apdex compresses latency into a single 0–1 satisfaction score against a target threshold.

check yourself

Quick quiz

Q1

Why report latency as p95/p99 instead of the average?

Q2

Roughly how much downtime per year does 99.99% ('four nines') availability allow?

follow the thread