System Atlas
the atlas

Where do we cache — and how it goes wrong

Caching lives at every layer. So do the ways it can bite you.

the gist

Caches sit at every hop — browser, CDN, gateway, application, and database. The wins are huge, but stale data, thundering herds, cache stampedes and the trio of penetration, breakdown and avalanche are real failure modes you design against.

ANALOGY // A library front desk keeps the most-requested books on a nearby shelf. Problems appear when the shelf copy is outdated, everyone asks for a book that doesn't exist, or every book's 'return by' date lands on the same day.
living diagram01 / 04
missClientCDNApp serverRedis cacheDatabase
01

Layers of cache

A cache can live at every hop between the user and your data — the browser, a CDN, the load balancer, an application cache like Redis, and the database's own buffer pool. Each layer is progressively more expensive to reach, so the ideal request short-circuits at the earliest layer that has a fresh copy. Thinking in layers is what lets you decide where a given piece of data should be cached.

02

Cache hit

On the happy path the application cache holds the value and answers directly in microseconds, with no round trip to the database at all. This is the case you design the whole system to maximize, because every hit is work the slow, hard-to-scale database never has to do. A high hit ratio is the single biggest lever on both latency and load.

03

Cache penetration

Penetration happens when clients repeatedly ask for keys that do not exist — the cache always misses and every request falls straight through to the database. Because there is nothing to cache, the protection is to cache the absence itself: store a null or empty marker (with a short TTL) so repeated lookups for missing data are absorbed at the cache. A bloom filter can reject known-missing keys even earlier.

04

Avalanche

An avalanche occurs when a large batch of keys was created together and therefore expires at the same moment, so a synchronized wave of misses slams the database all at once. The fix is to add a small random jitter to each key's TTL so expirations spread out over time instead of firing simultaneously. Guarding the rebuild with locks or serving stale data during the refresh smooths it further.

the full picture
01

Why we cache at all

A cache is a small, fast store that keeps copies of data or computed results so future requests can be answered without redoing expensive work. That work might be a slow database query, a call to another service, or a heavy computation. The economics are lopsided in caching's favor: reading from an in-memory cache like Redis takes microseconds, while a database query can take milliseconds and a cross-service call much longer, so a cache that satisfies even 90% of reads removes the vast majority of load from your slowest, hardest-to-scale components.

The guiding principle behind every cache is locality: real workloads are not uniform. A tiny fraction of the data — a viral tweet, a popular product, a logged-in user's own profile — accounts for a huge fraction of the requests. Caching exploits that skew by keeping the hot minority close and cheap, and it is one of the highest-leverage optimizations in all of systems design. The cost of that leverage is that you now have two copies of the truth, and keeping them agreed is where the difficulty lives.

02

Caches live at every layer

Caching is not one box; it is a series of them stacked between the user and the source of truth, each catching requests before they travel further and cost more:

  • Browser cache — the client stores assets and responses locally, so no network call happens at all.
  • CDN — static content is cached in edge locations physically near users, cutting latency and origin load.
  • Load balancer / gateway — can cache common responses right at the entry point.
  • Application cache — a shared in-memory store like Redis or Memcached holding query results, sessions and computed objects.
  • Database buffer pool — the database itself keeps hot pages in memory.

The mental model is that a request should short-circuit at the earliest layer that has a fresh answer. Every layer it skips is latency the user never pays and load your backend never sees.

03

Read and write strategies

How data gets into and out of the cache is a design decision with real freshness-versus-speed trade-offs.

For reads, cache-aside (lazy loading) is the most common: the application checks the cache, and on a miss it reads the database and populates the cache itself. It only ever caches data that is actually requested, but the first request for anything is always a miss. Read-through hides this by letting the cache load from the database on a miss automatically.

For writes, the choice is about consistency. Write-through writes to the cache and database together, so the cache is never stale but every write pays the database cost. Write-back writes to the cache immediately and flushes to the database asynchronously, which is very fast but risks losing data if the cache dies before the flush. Underneath all of these sits the eviction policy — usually LRU (least recently used) — deciding what to discard when the cache fills, and the TTL, which bounds how long any entry is trusted before it must be refreshed.

04

The classic failure modes

Caching's benefits come with a set of well-known ways it bites back, and naming them is a rite of passage in system design interviews.

  • Cache penetration — requests for keys that do not exist always miss and hammer the database directly. Defend by caching the null result, or using a bloom filter to reject known-absent keys up front.
  • Cache breakdown (the hot-key problem) — a single very popular key expires, and the flood of concurrent requests that were relying on it all rush to rebuild it at once. Defend with a mutex/lock so only one request rebuilds while others wait, or with logical (never-hard) expiry.
  • Cache avalanche — a large set of keys expires simultaneously (often because they were loaded together), producing a synchronized miss storm. Defend by adding random jitter to TTLs so expirations are spread out.

The common thread is stampedes: many clients discovering a miss at the same instant and all falling through to the source of truth together.

05

Staleness and practical guidance

The deepest problem in caching is not throughput, it is truth. The moment you copy data you invite it to disagree with its source. There is a famous quip that the two hardest things in computer science are naming things and cache invalidation, and it is only half a joke: deciding when a cached copy is no longer valid is genuinely hard, especially under concurrent writes.

In practice you manage staleness by choosing an acceptable window. Set TTLs to bound how stale data can get, and where correctness matters, actively invalidate or update the cache on write rather than waiting for expiry. Crucially, decide per data type how much staleness you can tolerate — a product's price must be near-exact, but a follower count that lags by thirty seconds harms nobody. Real production systems combine all of this: browser and CDN caching for static assets, a Redis tier with cache-aside reads and jittered TTLs for dynamic data, explicit invalidation on the writes that must stay correct, and monitoring on the hit ratio, because a quietly falling hit rate is often the first sign that your database is about to fall over.

what to remember
01

Cache layers: client → CDN → load balancer → application (Redis/Memcached) → database buffer pool.

02

Read patterns: cache-aside (lazy), read-through, write-through and write-back each trade freshness for speed.

03

Cache penetration: queries for data that doesn't exist bypass the cache and hammer the DB — cache the null.

04

Cache avalanche: many keys expire at once; stagger TTLs to avoid a synchronized miss storm.

05

Cache breakdown: one hot key expires and a flood rebuilds it at once — use a lock or logical expiry.

check yourself

Quick quiz

Q1

How do you defend against cache penetration?

Q2

What causes a cache avalanche?

follow the thread