How Apache Kafka works
A distributed commit log that decouples who produces data from who consumes it.
Kafka is a distributed, append-only commit log for streaming. Producers publish messages to topics; each topic is split into ordered, replicated partitions spread across brokers; consumers in consumer groups read partitions and track their own offsets. This decouples producers from consumers, buffers traffic spikes, and lets consumers replay history.
Produce
Producers publish messages to a named topic, and crucially they don't know or care who will eventually read them — Kafka simply appends each message to the end of the log. This decoupling means you can add new consumers later without touching the producer at all. The producer's job ends the moment the write is acknowledged as durably stored.
Partition & append
A topic is split into partitions, and each message is appended to the tail of exactly one partition, chosen by its key (or round-robin if none). Within a partition messages get a strictly increasing offset, giving ordered, append-only streams that can be written and read in parallel. Partitioning is the unit of both ordering and scale.
Replicate
Each partition is replicated across several brokers: one leader handles reads and writes while followers copy its log. If the leader's broker fails, an in-sync replica is promoted to leader so no acknowledged messages are lost and the stream stays available. Replication is what turns a log into durable, fault-tolerant storage.
Consume by offset
Consumers join a consumer group, and Kafka assigns each partition to exactly one member so the group divides the work. Each consumer tracks an offset — its committed position in a partition — which lets it read at its own pace, resume after a crash, and even rewind to replay history. Because reads don't delete data, many independent groups can consume the same topic simultaneously.
The problem: too many wires between services
Imagine a modern application where many services need the same events. A user places an order, and suddenly the billing service, the inventory service, the shipping service, the analytics pipeline, and the email service all need to know. The naive approach is to have the order service call each of them directly. This works for three services and collapses at thirty.
The result is a tangle of point-to-point connections — an N×M web where every producer must know every consumer's address, format, and availability. Add a new consumer and you have to modify the producer. If a downstream service is slow or down, the producer blocks or drops data. And when traffic spikes, the fast producer overwhelms the slower consumers because they are directly coupled. Systems built this way are brittle and impossible to evolve independently.
Kafka's answer is to put a durable buffer in the middle. Producers write their events to Kafka and are done; they never call consumers directly. Consumers read from Kafka whenever they are ready, at their own pace. The producer no longer knows or cares who is listening. This inversion — from a mesh of direct calls to a shared log everyone reads and writes — is the whole reason Kafka exists, and it is why it sits at the heart of so many event-driven architectures.
The core idea: a distributed, append-only commit log
To understand Kafka you must understand one data structure: the log. Not "log" as in application error messages, but the log as a fundamental append-only sequence — the same structure databases use internally for their write-ahead logs.
A log has two rules that make it powerful precisely because it is so simple:
- Append-only. New records go on the end, and only the end. Existing records are never modified in place and never inserted in the middle.
- Ordered and immutable. Each record gets a sequential number (its offset) that never changes. Record 5 is forever record 5, sitting between 4 and 6.
Because records are immutable and identified by a stable offset, reading is just "give me everything from offset N onward," and a reader's entire position is captured by a single number it remembers. There are no complex locks, no in-place updates to coordinate — writers append, readers scan.
Kafka takes this log and makes it distributed and durable: the log is spread across a cluster of servers, persisted to disk, and replicated for safety. Appending to the end of a file is one of the fastest things a computer can do (sequential disk writes are near-RAM speed), which is a big part of why Kafka can handle millions of messages per second. Everything else — topics, partitions, consumer groups — is machinery layered on top of this one humble, brilliant idea.
Producers, topics, partitions, brokers, replication
The core vocabulary maps directly onto the log idea:
- Producer — a client that appends messages. It picks a topic and fires; once Kafka acknowledges the write, the producer's job is done.
- Topic — a named stream of records, like a table name or a category ("orders", "clicks"). It is the logical thing producers write to and consumers read from.
- Partition — the physical, ordered log a topic is split into. A topic isn't one log but several partitions, and this is the key to scale: partitions of one topic live on different servers and are written and read in parallel. A message's partition is chosen by its key (all events for the same customer key land in the same partition, preserving their order) or round-robin if there's no key.
- Broker — a single Kafka server. A cluster is many brokers, and partitions are spread across them so the load and storage are shared.
- Replication — each partition is copied to several brokers. One replica is the leader (it handles all reads and writes); the others are followers that stay in sync. If the leader's broker dies, an in-sync follower is promoted, so the data survives machine failure with no loss of acknowledged messages.
The crucial payoff to remember: ordering is guaranteed within a partition, not across a topic. If you need two events to be processed in order, they must share a key so they land in the same partition. This per-partition ordering is the precise trade Kafka makes to get its enormous parallelism.
Consumer groups, offsets, and replay
On the read side, the star concepts are the consumer group and the offset, and together they explain almost everything about how Kafka reads scale and recover.
A consumer group is a set of consumer instances that cooperate to read a topic. Kafka assigns each partition to exactly one consumer in the group, so the group divides the work: a topic with 6 partitions can be read in parallel by up to 6 consumers in a group. Add consumers (up to the partition count) and throughput rises; lose one and its partitions are automatically rebalanced onto the survivors. This is how consumption scales and self-heals.
The offset is each consumer's bookmark: the position up to which it has processed a partition. Consumers periodically commit their offset back to Kafka. Two profound capabilities fall out of this:
- Independent pacing and multiple readers. Because reading does not delete data (Kafka retains messages for a configured time or size, regardless of who read them), many groups can read the same topic completely independently. The billing group and the analytics group each have their own offsets and neither affects the other.
- Replay. An offset is just a number you can reset. Point a consumer back to offset 0 and it re-reads history — invaluable for reprocessing after a bug fix, backfilling a new service, or recovering from a crash by resuming exactly where it left off.
This is the sharp contrast with a traditional queue, where reading a message removes it. Kafka is a retained log you scan, not a mailbox you drain.
Guarantees, trade-offs, and real-world use
Kafka forces you to think about delivery semantics, because in a distributed system acknowledgements and crashes can overlap. The three levels:
- At-most-once — commit the offset before processing; if you crash mid-work the message is lost but never duplicated. Rarely what you want.
- At-least-once — process first, then commit; if you crash after processing but before committing, you'll re-read and reprocess the message. This is the common default, and it means consumers must be idempotent (processing the same message twice has the same effect as once).
- Exactly-once — Kafka can approximate this within its own boundaries using idempotent producers and transactions, but true end-to-end exactly-once across external systems is genuinely hard; most teams choose at-least-once plus idempotency instead.
The trade-offs are real. Kafka is operationally heavy — a cluster with brokers, replication, and (historically) ZooKeeper or now KRaft for coordination is not something you spin up casually. It optimises for high throughput over ultra-low latency, and its per-partition ordering means you must design keys carefully. It is overkill for a small app that just needs a simple task queue.
Where it shines: decoupling many services via events, buffering traffic spikes so a burst doesn't crush downstream consumers, feeding real-time analytics and stream processing, aggregating logs and metrics, and event sourcing where the log itself is the system of record. The mental model to keep: Kafka is a durable, replayable, append-only log that lets producers and consumers evolve independently — decoupling, buffering, and replay are the three gifts, and everything else is detail in service of them.
Kafka is an append-only log: messages are added to the end and never edited in place.
Producers publish to topics; a topic is a named stream split into partitions for parallelism.
Each partition is ordered and replicated across brokers, so order is kept and data survives failures.
Consumers belong to consumer groups; partitions are divided among group members to scale reads.
Each consumer tracks an offset — its position in a partition — which enables replay and independent pacing.
Quick quiz
What does tracking an offset let a Kafka consumer do?
Why does Kafka split a topic into partitions?
How Netflix built a distributed counter
Counting sounds trivial — until it has to be right at global scale.
A crash course on scalability
How one box on a laptop becomes a system that serves millions.
How Kubernetes works
You declare the desired state; the cluster relentlessly makes it real.