NotesSystem

Apache Flink

Also: Flink

A distributed engine for stateful computation over unbounded streams, in event time. Its defining choice is local keyed state made fault-tolerant by asynchronous barrier snapshots, which gives exactly-once state and, with transactional sinks, exactly-once output.

Design choices

  • Stateful Stream Processing: state is partitioned by key, held on the heap (the default) or in embedded RocksDB, or since 2.0 disaggregated to object storage (ForSt, still experimental).
  • Checkpoints: barriers flow with the data, operators align and snapshot, and recovery rewinds sources to the snapshot's offsets. Unaligned checkpoints (1.11) help under backpressure. Savepoints are user-owned snapshots for upgrades.
  • Event Time vs Processing Time and Watermarks: bounded-out-of-orderness strategies, idleness, alignment (1.15), allowed lateness and side outputs for late data.
  • Windowing: tumbling, sliding, session and global windows. SQL window TVFs: TUMBLE, HOP, CUMULATE, SESSION (1.19).
  • Stream Joins: interval, temporal (event-time, versioned), lookup and regular joins.
  • Exactly-Once Semantics: two-phase-commit sinks (since 1.4) driven by checkpoint completion, e.g. Kafka transactions.

Notable trade-offs

  • Exactly-once output adds latency (results are visible per checkpoint) and needs transactional sinks plus careful timeout settings.
  • Large state means long checkpoints and slow recovery. That is the motivation for incremental checkpoints and disaggregated state.
  • Watermarks are only as fast as the slowest input; idle partitions stall them.

Scale

  • Alibaba: 4 billion records per second at its 2020 Double 11 peak. Netflix: more than 30,000 Flink jobs (2026). King: 30 billion+ events a day (2017).
  • Releases: 2.0 (March 2025), 2.3 (June 2026).