System performance metrics every engineer should know
Averages lie; percentiles and the golden signals tell you the truth.
Performance boils down to a handful of numbers: latency (measured at percentiles, not averages), throughput, error rate, saturation, and availability, with Apdex rolling latency into a user-satisfaction score. The single most important idea is the difference between the average and the tail — your worst requests, captured by p95 and p99, are what actually shape user experience.
Latency
Latency is the time to serve a single request — the delay a user feels between asking and receiving. It's the most direct measure of responsiveness. But a single latency number is meaningless without knowing how it's distributed, which is why raw averages are a trap.
Percentiles and the tail
Report latency as percentiles: p50 (median) is a typical request, while p95 and p99 are the slow tail — the value 95% or 99% of requests come in under. The tail is what actually shapes experience, because at scale that 'rare' slow 1% happens to millions of users and often to your most active ones. Chasing the average hides your worst behaviour; watching p99 exposes it.
Throughput
Throughput is how much work the system does per unit time, usually requests per second (RPS) or queries per second. It measures capacity rather than speed — a system can be fast per request yet low-throughput, or high-throughput with mediocre latency. Together, latency and throughput describe how a system behaves as load rises.
Error rate
Error rate is the fraction of requests that fail — HTTP 5xx responses, timeouts, exceptions — usually expressed as a percentage of total traffic. It's the most direct signal of correctness and health; a rising error rate means users are hitting failures right now. It pairs naturally with latency, since a slow system often starts shedding errors under stress.
Saturation & utilisation
Saturation measures how full a resource is — CPU, memory, disk I/O, connection pools, queue depth — as a fraction of its maximum. It's the leading indicator of trouble: latency and errors usually stay flat until a resource nears 100%, then degrade sharply. Watching saturation tells you how much headroom remains before performance falls off a cliff.
Availability
Availability is the proportion of time the system is up and serving, expressed in 'nines'. 99.9% ('three nines') allows about 8.7 hours of downtime a year; 99.99% allows about 52 minutes; 99.999% about 5 minutes. Each extra nine is roughly ten times harder and costlier to achieve, so targets are chosen deliberately against what the business actually needs.
Apdex
Apdex (Application Performance Index) compresses latency into a single 0–1 satisfaction score against a target threshold T. Requests under T count as satisfied, those between T and 4T as tolerating, and slower ones as frustrated. It's a convenient executive-friendly summary, though like any single number it should sit alongside the percentiles, not replace them.
Why you need metrics at all
"Is the system fast?" is not an answerable question until you make it measurable. Performance work lives or dies on metrics — precise, quantitative signals that tell you how a system is actually behaving rather than how it feels. Without them you're guessing: you can't tell whether a change helped, whether you're near a cliff, or whether the outage you're firefighting is latency, errors, or capacity. With them, you can set targets, alert before users notice, and reason about trade-offs.
The good news is that the space is small. A handful of numbers — latency, throughput, error rate, saturation, and availability — capture the vast majority of what you need, and Google's "four golden signals" (latency, traffic, errors, saturation) and the related RED and USE methods are just curated subsets of them. The single most important skill isn't memorising the list, though; it's learning to *interpret* these numbers correctly, and the biggest interpretation mistake, by far, is trusting averages.
Latency, and why the average lies
Latency is the time to serve a single request — the delay between a user asking and receiving. It's the most direct measure of responsiveness, and the instinct is to summarise it with an average: "our average response time is 100 ms." That instinct is dangerous.
The problem is that latency distributions are almost never symmetric; they have a long tail of slow requests. Imagine 99 requests at 50 ms and one at 5000 ms — the average is a reassuring 99 ms, completely hiding that one user waited five seconds. Averages smear your worst behaviour into your best and make it vanish.
The fix is percentiles. The p50 (median) is a genuinely typical request. The p95 and p99 describe the tail: p99 = 500 ms means 99% of requests were faster than 500 ms and the slowest 1% were worse. Percentiles can't be averaged away because they report actual experienced values at the edges of the distribution. Whenever you see a latency figure with no percentile attached, treat it with suspicion.
The tail is the experience
Why obsess over the slowest 1%? Because at scale, "rare" events are constant, and they concentrate on the people who matter most. If a single page load makes 100 backend calls, the chance that *at least one* of them hits your p99 latency is enormous — so your p99 per-call latency becomes something close to your *typical* page latency. Slow tails compound.
Worse, tail latency correlates with load. Your heaviest users — the ones with the fullest carts, the most data, the most engagement — tend to trigger the slowest requests, so the tail disproportionately punishes your best customers. This is why large engineering organisations set service-level objectives on p99 (and even p99.9), not on the mean. "Tail latency" is not a niche concern for perfectionists; it is the metric that most directly maps to whether the product feels fast. The discipline is simple to state and hard to live by: optimise and alert on the tail, because that's where your users actually are.
Throughput, errors and saturation
Latency describes *how fast*; three more signals describe *how much*, *how healthy*, and *how full*.
- Throughput is the amount of work done per unit time — requests per second (RPS), queries per second, transactions per second. It measures *capacity*, which is distinct from speed: a system can serve each request quickly yet handle few in parallel, or handle a torrent at mediocre per-request latency. Latency and throughput together describe how a system behaves as load climbs, and they usually trade off — push throughput toward the limit and latency rises.
- Error rate is the fraction of requests that fail — 5xx responses, timeouts, exceptions — as a percentage of total traffic. It's the most direct correctness signal: a spike means users are hitting failures *now*.
- Saturation (utilisation) is how full a constrained resource is — CPU, memory, disk I/O, connection pools, queue depth — relative to its maximum. It's the crucial *leading* indicator: latency and errors often stay flat and healthy right up until a resource approaches 100%, and then degrade sharply. Watching saturation tells you how much headroom you have before that cliff, so you can scale *before* the numbers that users feel go bad.
Availability, Apdex and using them together
Availability is the proportion of time a system is up and correctly serving requests, conventionally expressed in nines. The scale is punishing: 99% ("two nines") permits ~3.65 days of downtime a year, 99.9% ~8.7 hours, 99.99% ~52 minutes, and 99.999% ("five nines") only ~5 minutes. Each additional nine is roughly ten times harder and more expensive to achieve — requiring redundancy, failover, and relentless operational rigour — so availability targets are a deliberate business decision, not a badge to maximise blindly. A five-nines target for a photo-sharing hobby app would be a waste; for a payments processor it's table stakes.
Apdex (Application Performance Index) rolls latency into a single 0–1 user-satisfaction score against a target threshold T: requests faster than T are "satisfied," those between T and 4T are "tolerating," and slower ones are "frustrated," combined into one number. It's handy for dashboards and non-engineers, but like any single-number summary it can hide detail, so it should complement the percentiles rather than replace them.
The practical way to use all of this is layered. Start with the golden signals — latency (at percentiles), traffic/throughput, errors, and saturation — as your always-on health view. Set SLOs on the tail latency and error rate that reflect real user expectations, alert on saturation to get ahead of trouble, and track availability against a target justified by the business. Above all, keep asking of every number: is this the average smoothing over a painful tail? The engineers who reason well about performance are the ones who never stop distinguishing the typical case from the worst case.
Latency is how long one request takes; always report percentiles (p50, p95, p99), never just the average.
The tail (p99) matters most: at scale, a slow 1% of requests hits users constantly and drags experience down.
Throughput (requests per second) measures capacity — how much work the system handles per unit time.
Availability is uptime measured in 'nines': 99.9% is ~8.7h/yr down, 99.99% is ~52min/yr.
Error rate (% of failed requests) and saturation (how full a resource is) reveal health and headroom.
Apdex compresses latency into a single 0–1 satisfaction score against a target threshold.
Quick quiz
Why report latency as p95/p99 instead of the average?
Roughly how much downtime per year does 99.99% ('four nines') availability allow?
Latency vs throughput
How fast one thing finishes versus how many things finish per second.
How to load websites at lightning speed
The bytes you never send, the trips you never make: a tour of every layer that makes a page feel instant.
How to debug a slow API
Measure before you guess: a disciplined walk down the request path to find where the milliseconds go.
Where do we cache — and how it goes wrong
Caching lives at every layer. So do the ways it can bite you.