Stream Processing Message Semantics: Exactly-Once vs At-Least-Once

Explain the three message delivery semantics in stream processing.

At-Most-Once

Messages may be lost but never duplicated. Suitable when loss is acceptable (e.g., real-time metric monitoring).

At-Least-Once

Messages are never lost but may be duplicated. The most common default. Downstream systems must be idempotent to handle duplicates.

Exactly-Once

Each message is processed exactly once — no loss, no duplicates. Most complex to implement; highest performance cost.

Exactly-Once in Kafka

• Producer side: Idempotent producer (enable.idempotence=true) + Transactions API for atomic writes.
• End-to-end: Combined with Kafka Streams or Flink using 2PC or checkpoint mechanisms for end-to-end exactly-once.

Practical Choice

At-Least-Once + idempotent downstream satisfies most use cases with better performance. Only high-precision requirements like financial transactions justify the cost of Exactly-Once.