System Atlas
the atlas

Batch vs stream processing

Process the data after it piles up, or the instant it arrives.

the gist

Batch processing runs over bounded datasets on a schedule — high throughput but higher latency, the model behind Hadoop and Spark. Stream processing handles unbounded data event-by-event as it arrives — low latency but harder to get right, the model behind Flink and Kafka Streams. Windowing and exactly-once semantics are the machinery that makes streaming reliable.

ANALOGY // Batch is a factory that ships one big truckload of finished goods each night; streaming is a conveyor belt that finishes and ships each item continuously as it comes off the line.
living diagram01 / 05
Batch: bounded datafinite, known dataset
Stream: unbounded datanever-ending flow
Batch latencyminutes to hours
Stream latencymilliseconds to seconds
Batch throughputhuge bulk volume
Stream throughputsteady per-event
Batch complexityrerun on failure
Stream complexitywindows + exactly-once
Batch works on finite data; streams are endless.
01

Bounded vs unbounded data

Batch processing operates on a bounded dataset — a finite, known chunk like 'yesterday's transactions' — that has a clear beginning and end. Stream processing operates on unbounded data — an endless flow of events, like clicks or sensor readings, that never completes. This distinction in the shape of the input is the root of every other difference between the two models.

02

Latency: periodic vs continuous

Batch jobs run on a schedule or when enough data accumulates, so results lag the events by minutes to hours — you learn what happened after the batch completes. Stream jobs react within milliseconds to seconds of each event arriving, so insights are effectively real-time. If you need to act the moment something happens, batch latency is disqualifying.

03

Throughput and efficiency

Batch shines at raw throughput: processing huge volumes together amortizes overhead, allows heavy optimization, and is simple to reason about because the whole dataset is available at once. Streaming trades some of that bulk efficiency for immediacy, doing steady per-event work rather than one enormous pass. For massive periodic computations, batch remains the most cost-effective choice.

04

Windowing bounds the endless

Since a stream never ends, you cannot 'sum all events' — so aggregations happen over windows that slice the flow by time or count. Tumbling windows are fixed and non-overlapping, sliding windows overlap, and session windows group bursts of activity separated by gaps. Windowing is what turns an infinite stream into finite, answerable questions like 'orders in the last five minutes.'

05

Exactly-once and correctness

Streams fail mid-flight, and a naive retry can double-count events, so serious stream engines provide exactly-once semantics through periodic checkpoints and idempotent writes. On failure the job restores its last consistent checkpoint and replays only what's needed, so each event affects the result once. This machinery is why streaming is harder to build than batch, where a failed job simply reruns from the start.

the full picture
01

Two paradigms for processing data

Every data pipeline has to answer one question: do we process the data *after* it accumulates, or *as* it arrives? Those two answers are batch processing and stream processing, and they represent fundamentally different ways of thinking about data and time.

Batch processing collects data into a finite, bounded set and processes it all at once, usually on a schedule — every night, every hour. Stream processing handles data as an unbounded, continuous flow, processing each event (or tiny group of events) within moments of its arrival. The single deepest distinction is the *boundedness* of the input: batch assumes the data has an end you can wait for, streaming assumes it never does. Almost every practical difference — latency, throughput, complexity, tooling — descends from that one assumption.

02

How batch processing works

In the batch model you have a dataset with known boundaries — "all of yesterday's transactions," "this month's logs" — and you run a job that reads the whole thing, transforms it, and writes results. Because the entire input is available up front, batch jobs can be heavily optimized: they can sort, group, and make multiple passes, and they amortize fixed overhead across an enormous volume, which makes them extremely efficient per record.

The canonical lineage is Hadoop MapReduce and, more commonly today, Apache Spark, which run computations across clusters over data sitting in a data lake, warehouse, or distributed filesystem. Batch's great virtues are throughput and simplicity: it excels at crunching very large volumes, and its failure model is easy — if a job dies, you just rerun it from the start, because the input is fixed and reprocessing is deterministic. The cost is latency: results are only as fresh as the last run, so you inherently learn about events after the fact.

03

How stream processing works

Stream processing flips the model. Data arrives continuously as an endless sequence of events — user clicks, IoT readings, payments, log lines — and the engine processes each one as it comes, emitting results within milliseconds to seconds. The point is immediacy: you can detect and react to things *as they happen* rather than waiting for a scheduled job.

The tooling reflects this. A durable log like Apache Kafka transports the events, and engines like Apache Flink, Kafka Streams, or Spark Structured Streaming compute over them in flight. The hard part is that the input never ends and events can arrive late or out of order, so a streaming engine must maintain running state (running counts, joins, aggregations) and manage time carefully — distinguishing *event time* (when something actually happened) from *processing time* (when the engine saw it). This continuous, stateful, time-aware nature is what makes streaming powerful and also what makes it harder to build than batch.

04

Windowing and exactly-once — the streaming machinery

Two ideas make streaming tractable and correct.

  • Windowing. You cannot compute "the total of all events" over an infinite stream, because it never completes. So streaming aggregations operate over windows that carve the flow into finite chunks. *Tumbling* windows are fixed-size and non-overlapping (each minute, separately). *Sliding* windows overlap (the last 5 minutes, recomputed every minute). *Session* windows group bursts of activity separated by gaps of inactivity. Windows are how you turn an unbounded stream into answerable questions like "purchases in the last ten minutes."
  • Exactly-once semantics. Streams run forever, so failure is not an exception, it's a certainty — and a naive retry after a crash can process the same event twice, double-counting money or metrics. Serious engines provide exactly-once processing using periodic checkpoints (consistent snapshots of the job's state and stream position) plus idempotent or transactional writes. When a job fails it restores its last checkpoint and replays from there, so every event affects the result exactly once. Weaker guarantees — *at-most-once* (may lose data) and *at-least-once* (may duplicate) — are cheaper but riskier.
05

Choosing between them, and the modern blend

The decision is about latency tolerance versus simplicity and cost:

  • Choose batch when data naturally arrives in bulk, some delay is fine, and you value throughput and operational simplicity — nightly analytics, billing runs, large ETL jobs, model training over historical data.
  • Choose streaming when acting quickly is the whole point — fraud detection that must block a transaction now, real-time dashboards and monitoring, alerting, live recommendations, or feeding fresh features to online models.

In reality the line has blurred. Frameworks increasingly treat batch as a special, bounded case of streaming, and architectures often run both together. The old Lambda architecture paired a batch layer (accurate, slow) with a speed layer (fast, approximate) and merged them; the newer Kappa architecture argues you can do everything with one streaming pipeline and simply reprocess history by replaying the log. The pragmatic takeaway is to match the tool to the *freshness* the business actually needs: don't pay streaming's complexity tax for a report nobody reads until morning, and don't force a batch job to fake real-time when the value is in reacting instantly.

what to remember
01

Batch = bounded, finite data processed periodically; optimized for throughput, tolerant of latency.

02

Stream = unbounded, never-ending data processed as it arrives; optimized for low latency.

03

Batch tools: Hadoop MapReduce, Spark. Stream tools: Flink, Kafka Streams, Spark Structured Streaming.

04

Streams have no natural end, so aggregations use windows (tumbling, sliding, session) to bound time.

05

Exactly-once processing (via checkpoints and idempotency) prevents double-counting when a stream job fails and retries.

06

Choose batch for large periodic reports; stream for real-time detection, dashboards, and alerts.

check yourself

Quick quiz

Q1

Why do streaming aggregations need windows?

Q2

Which workload is the best fit for batch processing?

follow the thread