advanced

Kafka delivery semantics

Explain at-most-once, at-least-once, effectively-once, transactions, idempotent producers, and why consumers still need safe side effects.

At-most-once: commit offset before process — may lose messages. At-least-once: process then commit — duplicates on failure. Effectively-once combines idempotent producer, transactions, and idempotent consumer side effects. Kafka does not magically dedupe business logic.

Trade-off: stronger guarantees cost latency, complexity, and broker resources.

On interviews: map semantics to payment processing; why consumers still need inbox or idempotency keys.

Common pitfalls: claiming exactly-once without transactional boundaries; idempotent producer but non-idempotent consumer DB writes.

Checklist:

State chosen semantics per topic explicitly.
Pair broker settings with application idempotency.
Test failure between process and commit.
Document duplicate handling for operators.