System Atlas
the atlas

Latency vs throughput

How fast one thing finishes versus how many things finish per second.

the gist

Latency is the time a single operation takes — the delay. Throughput is how many operations complete per unit time — the rate or bandwidth. They're independent: you can have high throughput with high latency (a wide but slow pipe) or low latency with low throughput. Optimizing one doesn't automatically fix the other.

ANALOGY // A highway: latency is the travel time for a single car end to end, while throughput is the number of cars that pass per hour. Adding lanes raises throughput without making any one car's trip faster.
living diagram01 / 04
Latencytime per operation (delay)
Throughputoperations per second (rate)
Two independent axes: delay versus rate.
01

Two different questions

Latency asks 'how long does one operation take?' while throughput asks 'how many operations finish per second?' They answer genuinely different concerns and are measured in different units — time versus rate. Confusing them is one of the most common performance-reasoning mistakes, because improving one says nothing about the other.

02

Latency is delay

Latency is the end-to-end time for a single operation — one request, one query, one car's trip down the highway — usually measured in milliseconds. Lower latency feels more responsive to a user. It's bounded from below by hard physics like the speed of light over distance, so no amount of extra capacity can push it below that floor.

03

Throughput is rate

Throughput is the volume of work completed per unit time — requests per second, bytes per second, the effective bandwidth of the system. Higher throughput means more total work gets done, and unlike latency you can usually raise it by adding parallel capacity. It's about aggregate volume, not the experience of any single request.

04

They're independent

A wide, slow pipe demonstrates the independence: it can carry enormous throughput while each individual item still takes a long time to arrive, so both are high at once. Adding lanes to a highway lifts throughput without shortening any single car's trip, and shrinking a request's latency does nothing for how many you can serve. So you must decide which metric actually matters and tune deliberately for it.

the full picture
01

Two words people constantly conflate

Ask why a system feels slow and you'll get answers mixing up two completely different quantities. Latency is how long a single operation takes — the delay between asking and receiving. Throughput is how many operations complete per unit of time — the rate. Latency is measured in units of time (milliseconds); throughput in units of things-per-time (requests per second, gigabits per second).

The classic picture is a highway. Latency is how long it takes one car to drive from one end to the other. Throughput is how many cars pass a point per hour. These describe the same road but answer different questions, and — this is the whole lesson — they are largely independent. You can make the road move more cars per hour without any single car arriving sooner, and you can make each car faster without moving more cars. Keeping them separate in your head is the foundation of reasoning about performance.

02

Why they're independent

The intuition that "faster = more" fails because latency and throughput have different bottlenecks. Latency is dominated by the length of the critical path: distance (the speed of light adds roughly 1 ms per 100 miles each way), the number of sequential hops, disk seeks, and processing steps that must happen one after another. Throughput is dominated by capacity and parallelism: how many requests you can process at once, how wide the pipe is, how many workers you have.

Because of that, you can raise throughput by doing more things *in parallel* without shortening any single thing. The vivid example is the wide, slow pipe: a satellite link or a truck full of hard drives can move enormous volumes of data (huge throughput) while each bit takes a long time to arrive (terrible latency). Conversely a tiny, ultra-low-latency link can be quick per request yet unable to carry much total volume. High throughput and high latency coexisting is not a paradox — it's the normal state of many real systems.

03

The link between them: concurrency

The two aren't *totally* unrelated — they're tied together by how much work is in flight at once. A useful mental model is Little's Law: the average number of items in a system equals the arrival rate multiplied by the average time each spends inside (concurrency = throughput × latency). Rearranged, throughput = concurrency ÷ latency.

That equation captures a lot of real behavior. If each request takes 100 ms and you can only handle one at a time, your ceiling is 10 requests per second no matter how fast the network is — so to raise throughput you either cut latency or increase concurrency (more threads, connections, or servers). It also explains why bandwidth alone doesn't fill a high-latency link: a single TCP connection can only have one "window" of unacknowledged data in flight, so on a long-delay path (high latency), one connection can't come close to saturating the available bandwidth. You need many parallel connections — more concurrency — to convert the raw capacity into actual throughput.

04

Tail latency and why averages lie

When you measure latency, the *average* is almost useless on its own. What users feel is the distribution, and specifically the tail — the slowest few percent, reported as percentiles like p95, p99, and p999 (the 99th percentile is the value 99% of requests come in under). A service with a great average and an ugly p99 still delivers a bad experience to a meaningful fraction of requests.

Tail latency matters more than intuition suggests because of fan-out. If rendering one page requires 100 internal service calls in parallel and the page can't finish until the slowest returns, then even a rare 1-in-100 slow call means most pages hit that slow path. So a p99 that seems like an edge case becomes the *typical* experience at scale. This is why serious systems track and optimize percentiles, not means — cutting the tail (via timeouts, retries, hedged requests, or removing a slow dependency) often improves perceived performance far more than shaving the average.

05

Designing for the metric that matters

Because you often can't maximize both, the practical work is deciding which one your use case actually cares about — and the answer differs sharply by workload. User-facing, interactive paths live and die on latency: a checkout page, an autocomplete, an API call a human is waiting on. Batch and bulk workloads care about throughput: a nightly analytics job, a data-pipeline backfill, video encoding — nobody is watching any single item finish, so you optimize total volume even at the cost of per-item delay.

The techniques diverge accordingly. To cut latency: cache aggressively, move computation and data closer to the user (CDNs, edge), reduce round trips, precompute, and trim the critical path. To raise throughput: batch operations, add parallel workers, buffer and pipeline, and scale horizontally. Some moves even trade one for the other — batching amortizes fixed cost to lift throughput but makes each item wait for the batch, adding latency. The discipline is to name your target explicitly, measure the right thing (percentiles for latency, sustained rate for throughput), and accept that a system tuned for one is usually not tuned for the other.

what to remember
01

Latency measures delay: the time for one operation to complete, measured in units like milliseconds.

02

Throughput measures rate: operations completed per unit time, often called bandwidth.

03

They are independent axes — improving one can leave the other unchanged.

04

High throughput can coexist with high latency: a wide, slow pipe moves a lot but each item is slow.

05

Design deliberately for the one that matters: user-facing responsiveness wants low latency; batch jobs want high throughput.

check yourself

Quick quiz

Q1

Can a system have high throughput and high latency at the same time?

Q2

Which metric best captures user-facing responsiveness?

follow the thread