Kafka for event streaming and replay

Choose Kafka when you need durable, ordered event logs with replay, high throughput, and many independent consumer groups reading the same stream at their own pace. Event sourcing, analytics pipelines, audit trails, and cross-team integration via shared topics fit Kafka well.

Kafka retains messages by policy, not until processed — consumers track offsets. Partitioning provides parallelism while preserving order per key. Operational cost includes cluster sizing, rebalancing, schema registry, and consumer lag monitoring.

On interviews: contrast Kafka with RabbitMQ for a notification platform, explain replay value, and mention when Kafka is overkill.

Common pitfalls: using Kafka as a task queue with per-message ack semantics; too few partitions for throughput; no compaction strategy for changelog topics; ignoring consumer lag SLOs.

Checklist:

• Confirm need for replay and retained history.
• Size partitions for throughput and ordering keys.
• Plan schema evolution and registry.
• Monitor lag per consumer group.