Stream Processing · 2 of 4

Stream Processing

A computation that never finishes, over data that never stops and arrives out of order. How stream processors decide what time it is, when an answer is ready, and how they survive crashes without counting anything twice.

≈ 25 min

The whole idea

Press ← → to step, or Space to play and pause, whenever a diagram has focus.

From batches to streams

A batch job runs over a bounded input: yesterday’s files. It starts, reads everything, writes an answer and stops. Because the input is complete, “how many rides in Paris between 12:01 and 12:02?” has one right answer, and running the job twice gives it twice.

A stream job runs over an unbounded input, usually a partitioned log like Kafka, and never stops. That buys freshness: fraud flagged in seconds, a dashboard that’s live, a search index that’s current. It also removes the thing batch quietly relied on. There’s no moment when the data is complete. Events keep coming, some of them late, and the job can crash halfway through anything.

Google’s Dataflow Model paper (2015) put the response bluntly: “We as a field must stop trying to groom unbounded datasets into finite pools of information that eventually become complete.” It splits every streaming computation into four questions:

  • What is computed? (a count, a sum, a join)
  • Where in event time? (windows)
  • When in processing time are results emitted? (watermarks and triggers)
  • How do later results relate to earlier ones? (replace, accumulate, retract)

This lesson walks through them, then through what makes the answers survive failure: state, checkpoints and exactly-once.

Event time vs processing time

Every event carries two timestamps. Event time is when it happened, stamped by the device or service that produced it. Processing time is when the stream processor happens to see it. The gap between them, the skew, is never zero and never constant.

Two clocks

12:0012:0112:0212:0312:0412:0512:0012:0112:0212:0312:0412:05event time: when it happened →processing time: when it’s seen →ideal
Each dot is one event. Its distance above the diagonal is how late it was seen.

Skew comes from everywhere: mobile clients buffering offline, batching in producers, broker lag, a consumer restarting and replaying an hour of backlog. The last one matters most. When a job replays history, processing time is now for every event, so anything computed in processing time is meaningless on replay.

So the rule is simple: compute in event time whenever the answer is about the world (“rides per minute”, “sessions per user”). Use processing time only for things about the pipeline itself, like throughput, or where approximate and fast genuinely beats correct.Even event time has a trust problem. Device clocks drift and can be set wrong; some systems record the server’s receive time too and clamp wildly implausible timestamps. Event time is correct and reproducible. Its cost is that the processor must decide when it has probably seen everything for a given time. That is the watermark.

Windows

An aggregate over an infinite stream never finishes, so most stream computations slice event time into windows and compute per window, per key.

Three ways to slice time

12:0012:0112:0212:0312:0412:0512:06ClicksTumblingsize 1 minHoppingsize 2 min, every 1 minSessiongap 1 min
Tumbling and hopping windows are fixed by the clock; session windows are shaped by the data.
Window Shape Typical use Watch out for
Tumbling fixed size, no overlap per-minute metrics, billing periods a burst on a boundary splits in two
Hopping (sliding) fixed size, overlapping, advancing by a slide moving averages, “last 10 min, every minute” each event is in size/slide windows: more state, more output
Session per key, ends after a gap of inactivity user sessions, device activity bursts windows merge when a late event bridges a gap
Global one window per key, forever custom triggers, running totals never fires by default; state never expires

Flink offers all four in the DataStream API and as SQL table-valued functions: TUMBLE, HOP, CUMULATE (windows that grow within a period, like “revenue so far today, updated every minute”) and SESSION (added in Flink 1.19). Kafka Streams has the same family, plus sliding windows defined by the distance between records.

Watermarks

A window over 12:00–12:01 can’t emit a final count at 12:01 on the wall clock: events from 12:00:59 may still be in flight. The processor needs a signal that says “event time has reached 12:01 here”. That signal is a watermark: a timestamp flowing through the job asserting that no more events older than it are expected.

A watermark closes a window

max event time seen: —watermark = max − 10 sWindow 1Window 212:00:0012:00:3012:01:0012:01:3012:02:00event time →
Circles are numbered in arrival order and placed by event time. The watermark follows the largest timestamp seen, minus a fixed allowance.

In Flink the common strategy is forBoundedOutOfOrderness: the watermark is the largest event time seen so far minus a bound you choose (strictly, minus one more millisecond). A window fires when the watermark passes its end. The choice of bound is a dial between latency and completeness:

  • Too small, and the watermark runs ahead of real data. Results come out fast, and some events turn up behind it as late data.
  • Too large, and every result waits. A 10-minute bound means each per-minute count comes out at least 10 minutes after its minute ends.

Google’s MillWheel (2013) defined the low watermark recursively: a stage’s watermark is the minimum of its own oldest pending work and the watermarks of everything upstream. Flink does the same. An operator with several inputs takes the minimum of their watermarks, which is what makes it safe. It is also what makes it fragile.

One idle input stalls the job

The minimum is only as fast as the slowest input, including one that has gone quiet.

An idle Kafka partition is the classic way a Flink job silently stops producing results. A field write-up on exactly this lists idle partitions, missing watermark strategies and oversized out-of-orderness bounds as the usual causes. withIdleness(...) excludes a silent input from the minimum. The opposite problem, one fast source racing ahead while windows buffer its data waiting for a slow one, is what watermark alignment (Flink 1.15) addresses by pausing sources that drift too far ahead.

Late data

Heuristic watermarks are guesses, so some events will arrive behind them. The Dataflow paper is blunt that watermarks “are sometimes too fast” and “sometimes too slow”, and concludes that “watermarks alone are insufficient”. Real systems give you three tools:

  • Drop it. This is the default in both Flink (allowedLateness is 0) and Beam. MillWheel’s Zeitgeist pipeline dropped late records, about 0.001% of them.
  • Divert it. Flink’s sideOutputLateData sends late events to a separate stream for a repair job or an audit.
  • Accept it and refine. With allowed lateness, a window keeps its state past the watermark and re-fires with a corrected result for each late arrival. State is freed only when the watermark passes window end plus the allowance. Kafka Streams calls this the grace period. Since Kafka 3.0 (KIP-633) you have to choose it explicitly, instead of inheriting a silent 24-hour default.

Refinement is the “how” question. A trigger decides when a window emits: early, speculative results before the watermark, an on-time result at it, late updates after. An accumulation mode says what each emission means: a replacement for the last one, a running total, or a retraction of the old value plus the new one. Downstream must agree. A sink that appends every firing will double-count unless results are upserts keyed by window.

State

A stateless stream job, one that filters or reshapes each event, is easy: crash, restart, carry on. Almost everything interesting is stateful: counts per window, the last-seen value per key, buffered events waiting for a join partner, a fraud model’s running features. That state is partitioned by key along with the input, so each parallel instance owns the state for its keys and touches it locally, with no network round trip.

Flink keeps keyed state in a state backend:

  • HashMapStateBackend (the default) holds objects on the JVM heap: fastest, but limited by memory and GC.
  • EmbeddedRocksDBStateBackend keeps state in a local RocksDB, an LSM-tree, so it can exceed memory by orders of magnitude and supports incremental checkpoints (opt-in), which upload only new SSTable files.
  • ForSt (Flink 2.0, 2025) disaggregates state: its files live in remote object storage, with local disk as a cache. The VLDB 2025 paper on it reports up to 94% shorter checkpoints and up to 50% lower cost than Flink 1.20. The docs still mark it experimental.

Kafka Streams takes the other road. Each state store is a local RocksDB backed by a compacted changelog topic, so the log is the durable copy and a restarted instance rebuilds its store by replaying the changelog.

Checkpoints

Local state makes the job fast, and a crash deletes it. The job needs a way to save a consistent picture: the state of every operator and the input position that state corresponds to, as of the same instant. Pausing the whole job to take one would be correct and unbearably slow.

Barrier snapshots

Kafka sourcepartition 0offset 100Count by keystatea: 3b: 5Sinkemits countsDurable storageS3, GCS, HDFS…checkpoint 6 ✓ offset 100 · a: 3, b: 5
A checkpoint is a consistent cut through a running job: the source offset and the operator state describe the same point in the stream.

Flink’s answer, from a 2015 paper by Carbone and colleagues, is asynchronous barrier snapshotting, a variant of the Chandy–Lamport snapshot algorithm. The coordinator tells sources to record their offsets and inject a numbered barrier into their output. Barriers flow with the data. An operator with several inputs aligns: it holds back an input whose barrier has arrived until the barrier has arrived on all of them, then snapshots its state and forwards the barrier. When every task has acknowledged, the checkpoint is complete. Recovery loads the last complete checkpoint and rewinds every source to its recorded offsets.

The details that matter in production:

  • Backpressure slows checkpoints. Barriers queue behind data, so a backpressured job may take minutes to checkpoint, or time out. Unaligned checkpoints (Flink 1.11, 2020) let barriers overtake in-flight records and store those records as part of the checkpoint. They help with slow channels, not with slow state uploads.
  • At-least-once mode skips alignment. It has lower latency, but records can be replayed into state twice.
  • Savepoints are checkpoints you trigger and own, used to upgrade code, change parallelism or migrate a job. Checkpoints are owned by Flink and exist for recovery.
  • The checkpoint interval is a recovery-time dial. Everything since the last checkpoint is replayed after a crash.

Stream joins

Joins are where state gets big. Neither side of a streaming join waits for the other, so the join must remember one side until the matching record from the other arrives, possibly for a long time.

Joining two streams

A time bound tells the join when it can forget. Without one, it can never forget.
Join Example State it keeps Ends when
Stream–stream, windowed / interval clicks within 10 min of an impression both sides, for the interval the watermark passes the interval
Stream–table (temporal) each order priced at the exchange rate in force at its event time the latest versions of each table key old versions, once the watermark passes them
Lookup enrich each event from a database or cache, at processing time none, or a cache n/a, but results depend on when you looked
Table–table (regular) keep a joined view of two changelogs up to date both sides, forever never, unless you set a TTL

A stream–table join is the workhorse of enrichment. In Kafka Streams, a KStream–KTable join triggers only on stream records and looks up the table’s current value, or the value as of the record’s timestamp if the table is versioned. Flink’s event-time temporal join needs a versioned table with a primary key and watermarks on both sides, so that a replay gets the same answer. A processing-time lookup, by contrast, is non-deterministic: replay it next week and the rates have changed.

Flink’s docs say plainly that a regular SQL join keeps “both sides of the join input in Flink state forever”. table.exec.state.ttl bounds it, at the cost of missing matches older than the TTL. A Xebia case study describes regular joins that grew to an estimated 780 GB of state, with checkpoint timeouts that killed the job. Rewriting them as temporal joins brought checkpoints under 4 GB.

Exactly-once

Checkpoints give exactly-once state: after recovery, every input record has affected each operator’s state exactly once. They don’t stop a sink from writing the same output twice, because the sink wrote it before the crash and writes it again during the replay. End-to-end exactly-once needs the sink to take part in the checkpoint.

A transactional sink

Output becomes visible only when the checkpoint that covers it completes: two-phase commit, driven by checkpoints.

Flink added the TwoPhaseCommitSinkFunction pattern in 1.4 (2017). With Kafka as the output, the sink writes into a Kafka transaction. On the checkpoint barrier it flushes and pre-commits. When the coordinator reports the checkpoint complete, it commits. A crash aborts the open transaction. Consumers must read with isolation.level=read_committed (the default is read_uncommitted), or they see aborted records anyway.

Three things to know before turning it on:

  • Latency becomes the checkpoint interval. Downstream sees results only when a checkpoint commits.
  • Transaction timeouts. Flink’s Kafka sink defaults to a 1-hour transaction timeout, and Kafka brokers reject anything above transaction.max.timeout.ms (15 minutes by default). Raise the broker limit or lower the sink’s timeout, and keep it longer than your worst checkpoint-plus-recovery time, or a transaction can expire before it commits and its data is lost.
  • Only transactional or idempotent sinks qualify. Kafka, files committed by rename, databases with idempotent upserts. An email or an HTTP call can’t be rolled back. Make it idempotent with a key derived from the event, and accept at-least-once.

Kafka Streams gets the same result more simply, because input, state changelogs and output are all Kafka topics. With processing.guarantee=exactly_once_v2 (KIP-447 and KIP-732; the default is still at_least_once), each task’s consumed offsets, state-store updates and output records commit in a single Kafka transaction. Calls to anything outside Kafka are outside the guarantee.

Trade-offs

Choice You get You pay
Event time correct, reproducible results; replay works waiting for watermarks; late-data handling
Processing time lowest latency, no watermarks wrong under skew, different answers on replay
Small out-of-orderness bound fresh results more late data
Allowed lateness corrected results state held longer; downstream must accept updates
Heap state speed size limited by memory and GC
RocksDB / disaggregated state state far larger than memory serialization cost per access; remote I/O
Short checkpoint interval fast recovery, low exactly-once latency more checkpoint overhead
End-to-end exactly-once no duplicate output latency = checkpoint interval; transactional sinks only

In the real world

4 billionrecords per second through Alibaba's Flink at its Double 11 2020 peak (7 TB/s)
30,000Flink jobs Netflix runs across AWS regions (2026)
30 billion+events a day at King on Flink, in the 2017 VLDB paper
  • Alibaba, Singles’ Day. Alibaba’s real-time compute platform, built on Flink, peaked at four billion records per second (seven terabytes a second) during Double 11 in 2020, up from 2.5 billion the year before.
  • Netflix runs more than 30,000 Flink jobs, most of them generated by its Data Mesh platform rather than written by hand. Autoscaling that many jobs became a problem worth its own 2026 write-up.
  • King (Candy Crush) was one of the production users in the 2017 VLDB paper on Flink’s state management: over 30 billion events a day, with production deployments holding 100–500 GB of global state each.
  • Uber (2025) moved data-lake ingestion from batch to streaming: thousands of jobs, about 25% less compute than the batch equivalent, and freshness from hours to minutes.
  • Flink 2.x (2.0 in March 2025, 2.3 in June 2026) removed the DataSet API and legacy source and sink functions, added disaggregated state and asynchronous state access, and has been adding SQL-level features such as model inference (ML_PREDICT) and vector search.
  • Google Dataflow / Apache Beam carry the Dataflow Model’s triggers and accumulation modes directly. Beam’s defaults are to emit when the watermark passes the end of the window and to drop later data.

Pitfalls

  • Windowing in processing time by accident. It looks fine live and gives different answers on every replay.
  • An idle partition pinning the watermark. No errors, no results. Set idleness, then find out why the partition is quiet.
  • Unbounded state. Regular joins, global windows, session windows on keys that never go quiet, GROUP BY on high-cardinality keys with no TTL. State grows until checkpoints time out.
  • Appending refinements. With allowed lateness or early firings, one window emits several results. Write them as upserts keyed by window, or downstream double-counts.
  • “Exactly-once” through a non-transactional sink. Checkpoints make state exact. They don’t un-send an email.
  • Exactly-once consumers reading uncommitted data. Without read_committed, downstream sees records from aborted transactions.
  • Transaction timeouts shorter than recovery. A transaction that expires during a long restart is aborted by the broker, and its output is lost.

Recall

Go deeper

Keep going

The atomic notes for this lesson live in the knowledge graph:

Next on the map: Change Data Capture & Event Sourcing (databases as streams, streams as databases) and Streaming Systems as Architecture (Lambda, Kappa and unbundling the database).

Done with this lesson?

Mark it learned and the concepts it teaches will light up in your galaxy.

Next in Stream ProcessingChange Data Capture & Event Sourcing