Stream processing

Two clocks can describe one event

A purchase may happen at 10:02 and be processed at 10:07. Event time is 10:02, the time the event occurred at its source. Processing time is 10:07, the time the application processed it.

The choice changes the answer to a windowed query. An event-time report for 10:00 through 10:05 assigns the purchase to that interval even though processing started later. A processing-time report assigns records according to the clock of the machine executing the operation. Delays, outages, and replay can therefore change processing-time results.

Wrong "Real-time processing means event time and processing time are the same."

Right They name different moments. Low latency can make them close, but the definitions do not depend on that latency.

Windows make an unbounded stream finite enough to aggregate

An unbounded stream has no final record, so a request such as "count every event" has no natural completion point. A window gives the aggregation a bounded group.

• A tumbling window has a fixed size and does not overlap. With five-minute windows, an event belongs to one interval.
• A sliding window overlaps. An event can contribute to more than one window.
• A session window groups activity for a key and separates sessions with an inactivity gap. Its start, end, and size depend on the data.

Kafka Streams 4.3 uses a narrower term for one common overlapping form. Its hopping windows have a fixed size and advance interval. Other tools sometimes call these sliding windows. Kafka Streams reserves sliding window for windows based on differences between record timestamps. State the semantics when terminology could be ambiguous.

This Kafka Streams code declares a five-minute tumbling window with one minute of grace:

Duration windowSize = Duration.ofMinutes(5);
Duration gracePeriod = Duration.ofMinutes(1);

TimeWindows.ofSizeAndGrace(windowSize, gracePeriod);

A watermark says how far event time has progressed

An event-time operator needs a signal that tells it when to calculate a time window. In Flink, watermarks carry that progress through the stream. When a watermark advances beyond a window's end, the default event-time trigger can fire that window.

A watermark is a declaration that events at or before its timestamp should have arrived. Delayed events can still violate the declaration. Such a record is late when the watermark has already passed the record's event timestamp.

Wrong "A watermark proves that every older event arrived."

Right A watermark lets the application balance completeness against latency. Waiting longer can admit more delayed records, while advancing sooner produces results earlier.

Flink can keep a time window after its first firing for configured allowed lateness. An accepted late record can cause another firing and an updated result. Data that arrives after the allowed interval can be dropped or collected through a side output. The configuration makes those policies visible:

stream
    .keyBy(Order::customerId)
    .window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
    .allowedLateness(Duration.ofMinutes(1))
    .sideOutputLateData(lateOrders)
    .aggregate(new OrderCount());

Stateful operators remember previous records

map and filter can handle one record without remembering earlier input. Aggregations, windows, and joins need state. A per-customer count stores the current count; a windowed join stores records that might still find a match.

Flink partitions keyed state with the keyed stream. An operator can access the state for the current event's key. Kafka Streams tasks embed state stores for stateful processing, and those stores can be persistent or in memory.

State introduces two operational questions: how it recovers after failure, and when old state can be removed. Window grace or allowed lateness affects both result timing and how long window state must remain available.

Exactly once has a boundary

Flink combines checkpointed operator state with replay. On recovery it restores a completed checkpoint and resets a rewindable source to the recorded position. Kafka Streams can atomically commit input offsets, state-store updates, and output records written to Kafka when exactly_once_v2 is enabled.

Neither description means that an arbitrary external HTTP side effect joins the same transaction. Kafka Streams documents its guarantee around Kafka offsets, its state stores, and Kafka output topics.

Wrong "Exactly once means the code physically runs one time."

Right The processing system coordinates recovery so that committed state and supported outputs reflect each input once within the documented boundary. Retries may still occur during recovery.

Event time survives delay and replay

Suppose a device records an event at 09:58, connectivity returns at 10:06, and the processor handles the record at 10:07. Event time is 09:58. In this example, processing time is 10:07 and comes from the system clock of the machine executing the operation.

Event time is data-driven. It lets a historical replay use the timestamps carried by records instead of grouping them according to today's wall clock. Processing time needs no coordination between streams and machines, but Flink's documentation calls out its nondeterminism in distributed and asynchronous systems. Arrival speed, operator speed, and outages can affect the result.

Wrong "Use processing time for a replay because the records are being processed now."

Right Choose the clock that matches the question. A report about when activity occurred needs event time. A report about what the processor handled during each wall-clock interval uses processing time.

Kafka Streams 4.3 assigns every record a timestamp through TimestampExtractor. Those timestamps drive its data-dependent stream time, which advances when records arrive. The effective meaning depends on the extractor and Kafka record timestamp configuration.

Window choice changes membership and state

A tumbling window is fixed-size, non-overlapping, and gapless. With boundaries [12:00, 12:05) and [12:05, 12:10), a record at 12:05 belongs to the second window.

A general sliding window overlaps, so one record may appear in several window evaluations. Naming differs across systems. Kafka Streams calls fixed-size windows with a separate advance interval hopping windows. It uses sliding windows for fixed-size windows whose snapshots depend on differences between record timestamps.

Session windows have no fixed start and end. For each key, activity within the configured inactivity gap is merged into a session. An event outside that gap creates another session. A later event can bridge activity, so session implementations need merging behavior.

Kafka Streams exposes those choices explicitly:

// Fixed-size, overlapping: size 5 minutes, advance every minute.
TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5))
    .advanceBy(Duration.ofMinutes(1));

// Data-driven sessions separated by 5 minutes of inactivity.
SessionWindows.ofInactivityGapWithNoGrace(Duration.ofMinutes(5));

Windowing is stateful. A windowed aggregation stores an aggregate per key and window. A windowed join stores records that could still match. Wider windows, overlapping membership, and longer late-data allowances keep more state available.

Watermarks turn uncertainty into a policy

Flink represents event-time progress with watermarks. A Watermark(t) declares that event time has reached t and that records at or before t should have arrived. An event-time window can fire when the watermark passes its end.

Parallel input matters. Each parallel source subtask usually produces its own watermarks. An operator with multiple inputs tracks the minimum of its input event times. One idle or lagging input can therefore hold back downstream event-time progress.

Wrong "Set the watermark to the newest timestamp seen and every window will be current."

Right A record may arrive out of timestamp order. Watermark generation must represent the delay the application is prepared to tolerate. Advancing aggressively lowers result latency but classifies more records as late.

Late-data behavior needs a separate policy. In Flink, allowed lateness keeps a time window after its first event-time firing. An accepted late record may fire the window again. Once the watermark passes the window end plus allowed lateness, Flink removes the time window. Data discarded as late can be collected through a side output.

OutputTag<Order> tooLate = new OutputTag<>("orders-too-late") {};

SingleOutputStreamOperator<Count> counts = orders
    .keyBy(Order::customerId)
    .window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
    .allowedLateness(Duration.ofMinutes(1))
    .sideOutputLateData(tooLate)
    .aggregate(new OrderCount());

DataStream<Order> rejected = counts.getSideOutput(tooLate);

Kafka Streams uses stream time plus a window grace period. A record is discarded from a window when its timestamp belongs there but current stream time is greater than the window end plus grace.

State belongs to the operator and key

An operator is stateful when handling one event depends on information retained from earlier events. Flink keyed state is partitioned with the keyed stream and is accessible for the current key. This lets parallel instances work on different groups of keys while keeping state access local.

Kafka Streams provides state stores inside stream tasks. The DSL uses them behind aggregations and joins, and the Processor API permits custom processors to interact with stores. Kafka Streams also supports read-only external queries through Interactive Queries.

This distinction catches a common design error: a plain process-local map has no framework-managed recovery contract. Managed state gives the runtime the information it needs to snapshot, restore, or rebuild that state.

What exactly-once processing covers

Flink checkpoints record source positions and operator state. After failure, Flink restores the latest completed checkpoint and replays from its recorded source positions. The full guarantee requires a source that can rewind, and connector guarantees still matter.

Kafka Streams defaults to at_least_once. In Kafka 4.3, exactly_once_v2 requires brokers version 2.5 or newer. When enabled, Kafka Streams atomically commits consumed offsets, state-store updates, and writes to Kafka output topics.

Properties settings = new Properties();
settings.put(
    StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    StreamsConfig.EXACTLY_ONCE_V2
);

Wrong "Once exactly-once mode is enabled, calls to a payment API are included."

Right The Kafka Streams transaction covers the Kafka read-process-write path and state stores described by its contract. An external service needs a separate idempotency or coordination design.

Time semantics are part of the query

"Orders per five minutes" is incomplete until the clock is named. With event time, records are grouped by when the source says the event occurred. With processing time, they are grouped by the executing operator's system clock. Processing-time output can change when input delivery, operator throughput, or outages change.

Event time supports out-of-order input and historical replay, but a live unbounded stream cannot wait forever to learn that a window is complete. The system needs a progress estimate and a policy for records that violate it.

Kafka Streams 4.3 calls its record-driven progress stream time. It assigns timestamps through TimestampExtractor, and stream time advances when records arrive at a processor. Flink carries event-time progress explicitly in watermarks. The APIs differ, but both force the application to confront out-of-order data.

Wrong "Event time makes a stream deterministic regardless of watermark policy."

Right Event timestamps preserve the business clock. Finite waiting still creates a completeness and latency tradeoff when records can be delayed without a known bound.

Watermark progress is limited by the slowest input

In Flink, parallel source subtasks usually generate watermarks independently. An operator that consumes multiple inputs advances its event time to the minimum input event time. This preserves the progress claim across all inputs, while a lagging input can delay every downstream event-time window.

A watermark is not a completeness certificate. Flink defines late elements as records arriving after the event-time clock, signaled by watermarks, has passed their timestamp. Production policy therefore has at least two separate settings: how watermarks are generated and how a window handles late records.

For a five-minute window ending at 12:05 with one minute of allowed lateness, Flink retains the time window until the watermark passes 12:06. The default event-time trigger can first fire after the window end. An accepted late element can cause a late firing, so downstream consumers may see a revision rather than one immutable final row.

SingleOutputStreamOperator<Revenue> revenue = orders
    .keyBy(Order::region)
    .window(TumblingEventTimeWindows.of(Duration.ofMinutes(5)))
    .allowedLateness(Duration.ofMinutes(1))
    .sideOutputLateData(tooLate)
    .aggregate(new RevenueAggregate());

Kafka Streams expresses the related cutoff with grace. A record is discarded from a window when its timestamp belongs to the window and stream time is greater than window end plus grace. These mechanisms have related goals, but Kafka Streams stream time is not Flink's propagated watermark protocol.

Window names do not have universal semantics

Tumbling windows are the least ambiguous: fixed-size, non-overlapping, and gapless. Session windows are data-driven and separated by an inactivity gap. They are tracked per key and may merge.

The word sliding needs more care. Flink describes sliding windows as overlapping windows. Kafka Streams distinguishes hopping windows from sliding windows. A hopping window has a size and an advance interval, so its boundaries are periodic. Kafka Streams sliding windows are aligned to record timestamps and use timestamp differences to define membership.

// Kafka Streams 4.3: periodic overlapping windows.
TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(10))
    .advanceBy(Duration.ofMinutes(1));

// Kafka Streams 4.3: record-timestamp-based sliding windows.
SlidingWindows.ofTimeDifferenceAndGrace(
    Duration.ofMinutes(10),
    Duration.ofMinutes(2)
);

Interviewers use this vocabulary mismatch to see whether a candidate describes membership. "Each event belongs to every ten-minute window advanced once per minute" is precise even when two libraries choose different names.

Stateful operators define the recovery problem

An aggregation retains a partial result. A windowed join retains candidates that may still match. A session window retains per-key sessions that may merge. These operators need state whose partitioning follows the stream's keys.

Flink models keyed state as an embedded key-value store partitioned with keyed input. Key groups are its redistribution unit for keyed state. Kafka Streams embeds state stores in tasks and can use changelog streams to reconstruct tables and replicate state for fault tolerance.

Managed state lets the framework connect three operations during recovery: restore state, restore or reset source position, then resume processing. State lifetime is also a correctness setting. Removing window state before the accepted late-data interval ends prevents the operator from applying its stated policy.

Wrong "A local key-value store makes an operator stateful, so recovery is automatic."

Right The runtime must manage or reconstruct that store and coordinate it with input progress. A process-local map by itself supplies none of those guarantees.

Exactly once means atomic visibility inside a stated boundary

Flink checkpoints capture operator state together with positions in input streams. On failure it restores the latest completed checkpoint and replays from the recorded positions. A rewindable source is required for the mechanism's full guarantee. Source and sink connector guarantees determine the end-to-end result.

Kafka Streams 4.3 defaults to at-least-once. With exactly_once_v2, it uses Kafka transactions so input-offset commits, state-store updates, and Kafka output writes complete atomically. Failed work can be retried without exposing an extra committed result to consumers that use read_committed.

settings.put(
    StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
    StreamsConfig.EXACTLY_ONCE_V2
);

An HTTP call or an update to an unrelated database does not become part of that Kafka transaction because it appears inside a processor callback. The application needs a separately supported transactional or idempotent boundary for such effects.

Wrong "Exactly-once processing guarantees that user code executes once and no record is replayed."

Right Recovery may replay work. Exactly-once semantics coordinate state and supported outputs so the committed result reflects each input once within the documented boundary.

Checkpoint or transaction configuration alone is therefore insufficient evidence. A complete guarantee statement names the source, operator state, sink, late-data behavior, and what readers are allowed to observe.