advanced

Kafka offsets

Treat offsets as consumer progress that must be committed after safe processing, with replay and duplicate handling designed intentionally.

Offsets are per-partition positions consumers track — which messages were processed. Commits store progress in __consumer_offsets or external store. Commit after processing for at-least-once; commit before for at-most-once with loss risk on crash.

Trade-off: frequent commits increase overhead; rare commits increase duplicate work on restart.

On interviews: crash after process but before commit — what happens; exactly-once with transactions scope.

Common pitfalls: auto-commit with long processing losing messages on crash; manual commit out of sync with DB transaction.

Checklist:

Align commit timing with idempotent processing.
Use transactional consume-process-produce when needed.
Monitor commit rate and rebalance offset issues.
Document replay procedure from specific offset.