Logs & Streaming

Core idea:

• streaming correctness = explicit invariants
• implement state machines, not ad hoc logic

Shared State

• partition offsets
• consumer assignment
• producer epoch + sequence
• watermark and active windows
• last record index per key

Offset Invariant

• commit can advance only contiguously
• from committed + 1 upward
• stop at first gap

Assignment Invariant

• deterministic round-robin
• sorted consumers + sorted partitions
• rebalance = diff(current owner, new owner)

Watermark Invariant

• watermark = maxEventTime - allowedLateness
• drop event if eventTime < watermark
• emit window when windowEnd <= watermark

Idempotence Invariant

• per producer (or producer+partition) sequence tracking
• accept in-order only
• reject duplicates and gaps

Exactly-Once Invariant

• epoch fencing
• begin/send/offsets/commit/abort lifecycle
• commit applies records + offsets atomically
• offsets monotonic via max

Compaction Invariant

• keep latest record per key
• preserve retained record order
• tombstone kept if latest

Problems in this module

• offset-commit
• consumer-group-assign
• consumer-rebalance-plan
• streaming-metrics
• windowed-aggregation
• idempotent-producer
• log-compaction
• exactly-once-stream