Batch vs stream processing
Process the data after it piles up, or the instant it arrives.
Batch processing runs over bounded datasets on a schedule — high throughput but higher latency, the model behind Hadoop and Spark. Stream processing handles unbounded data event-by-event as it arrives — low latency but harder to get right, the model behind Flink and Kafka Streams. Windowing and exactly-once semantics are the machinery that makes streaming reliable.
Bounded vs unbounded data
Batch processing operates on a bounded dataset — a finite, known chunk like 'yesterday's transactions' — that has a clear beginning and end. Stream processing operates on unbounded data — an endless flow of events, like clicks or sensor readings, that never completes. This distinction in the shape of the input is the root of every other difference between the two models.
Latency: periodic vs continuous
Batch jobs run on a schedule or when enough data accumulates, so results lag the events by minutes to hours — you learn what happened after the batch completes. Stream jobs react within milliseconds to seconds of each event arriving, so insights are effectively real-time. If you need to act the moment something happens, batch latency is disqualifying.
Throughput and efficiency
Batch shines at raw throughput: processing huge volumes together amortizes overhead, allows heavy optimization, and is simple to reason about because the whole dataset is available at once. Streaming trades some of that bulk efficiency for immediacy, doing steady per-event work rather than one enormous pass. For massive periodic computations, batch remains the most cost-effective choice.
Windowing bounds the endless
Since a stream never ends, you cannot 'sum all events' — so aggregations happen over windows that slice the flow by time or count. Tumbling windows are fixed and non-overlapping, sliding windows overlap, and session windows group bursts of activity separated by gaps. Windowing is what turns an infinite stream into finite, answerable questions like 'orders in the last five minutes.'
Exactly-once and correctness
Streams fail mid-flight, and a naive retry can double-count events, so serious stream engines provide exactly-once semantics through periodic checkpoints and idempotent writes. On failure the job restores its last consistent checkpoint and replays only what's needed, so each event affects the result once. This machinery is why streaming is harder to build than batch, where a failed job simply reruns from the start.
Two paradigms for processing data
Every data pipeline has to answer one question: do we process the data *after* it accumulates, or *as* it arrives? Those two answers are batch processing and stream processing, and they represent fundamentally different ways of thinking about data and time.
Batch processing collects data into a finite, bounded set and processes it all at once, usually on a schedule — every night, every hour. Stream processing handles data as an unbounded, continuous flow, processing each event (or tiny group of events) within moments of its arrival. The single deepest distinction is the *boundedness* of the input: batch assumes the data has an end you can wait for, streaming assumes it never does. Almost every practical difference — latency, throughput, complexity, tooling — descends from that one assumption.
How batch processing works
In the batch model you have a dataset with known boundaries — "all of yesterday's transactions," "this month's logs" — and you run a job that reads the whole thing, transforms it, and writes results. Because the entire input is available up front, batch jobs can be heavily optimized: they can sort, group, and make multiple passes, and they amortize fixed overhead across an enormous volume, which makes them extremely efficient per record.
The canonical lineage is Hadoop MapReduce and, more commonly today, Apache Spark, which run computations across clusters over data sitting in a data lake, warehouse, or distributed filesystem. Batch's great virtues are throughput and simplicity: it excels at crunching very large volumes, and its failure model is easy — if a job dies, you just rerun it from the start, because the input is fixed and reprocessing is deterministic. The cost is latency: results are only as fresh as the last run, so you inherently learn about events after the fact.
How stream processing works
Stream processing flips the model. Data arrives continuously as an endless sequence of events — user clicks, IoT readings, payments, log lines — and the engine processes each one as it comes, emitting results within milliseconds to seconds. The point is immediacy: you can detect and react to things *as they happen* rather than waiting for a scheduled job.
The tooling reflects this. A durable log like Apache Kafka transports the events, and engines like Apache Flink, Kafka Streams, or Spark Structured Streaming compute over them in flight. The hard part is that the input never ends and events can arrive late or out of order, so a streaming engine must maintain running state (running counts, joins, aggregations) and manage time carefully — distinguishing *event time* (when something actually happened) from *processing time* (when the engine saw it). This continuous, stateful, time-aware nature is what makes streaming powerful and also what makes it harder to build than batch.
Windowing and exactly-once — the streaming machinery
Two ideas make streaming tractable and correct.
- Windowing. You cannot compute "the total of all events" over an infinite stream, because it never completes. So streaming aggregations operate over windows that carve the flow into finite chunks. *Tumbling* windows are fixed-size and non-overlapping (each minute, separately). *Sliding* windows overlap (the last 5 minutes, recomputed every minute). *Session* windows group bursts of activity separated by gaps of inactivity. Windows are how you turn an unbounded stream into answerable questions like "purchases in the last ten minutes."
- Exactly-once semantics. Streams run forever, so failure is not an exception, it's a certainty — and a naive retry after a crash can process the same event twice, double-counting money or metrics. Serious engines provide exactly-once processing using periodic checkpoints (consistent snapshots of the job's state and stream position) plus idempotent or transactional writes. When a job fails it restores its last checkpoint and replays from there, so every event affects the result exactly once. Weaker guarantees — *at-most-once* (may lose data) and *at-least-once* (may duplicate) — are cheaper but riskier.
Choosing between them, and the modern blend
The decision is about latency tolerance versus simplicity and cost:
- Choose batch when data naturally arrives in bulk, some delay is fine, and you value throughput and operational simplicity — nightly analytics, billing runs, large ETL jobs, model training over historical data.
- Choose streaming when acting quickly is the whole point — fraud detection that must block a transaction now, real-time dashboards and monitoring, alerting, live recommendations, or feeding fresh features to online models.
In reality the line has blurred. Frameworks increasingly treat batch as a special, bounded case of streaming, and architectures often run both together. The old Lambda architecture paired a batch layer (accurate, slow) with a speed layer (fast, approximate) and merged them; the newer Kappa architecture argues you can do everything with one streaming pipeline and simply reprocess history by replaying the log. The pragmatic takeaway is to match the tool to the *freshness* the business actually needs: don't pay streaming's complexity tax for a report nobody reads until morning, and don't force a batch job to fake real-time when the value is in reacting instantly.
Batch = bounded, finite data processed periodically; optimized for throughput, tolerant of latency.
Stream = unbounded, never-ending data processed as it arrives; optimized for low latency.
Batch tools: Hadoop MapReduce, Spark. Stream tools: Flink, Kafka Streams, Spark Structured Streaming.
Streams have no natural end, so aggregations use windows (tumbling, sliding, session) to bound time.
Exactly-once processing (via checkpoints and idempotency) prevents double-counting when a stream job fails and retries.
Choose batch for large periodic reports; stream for real-time detection, dashboards, and alerts.
Quick quiz
Why do streaming aggregations need windows?
Which workload is the best fit for batch processing?
How Apache Kafka works
A distributed commit log that decouples who produces data from who consumes it.
What is event sourcing
Instead of storing where you are, store how you got there.
Concurrency is not parallelism
Dealing with many things at once is a different idea from doing many things at once.
How data lake architecture works
One place to land any data now and decide what it means later.