advanced

Kafka retention

Set time or size retention by recovery, replay, audit, and storage needs rather than treating Kafka as either only a queue or a database.

Retention deletes old segments by time (retention.ms) or size (retention.bytes) per topic. Balance recovery needs, replay for new consumers, audit requirements, and disk cost. Unlimited retention turns Kafka into an expensive database.

Trade-off: long retention aids replay and debugging versus storage and compaction overhead.

On interviews: set retention for high-volume metrics versus financial audit events.

Common pitfalls: default retention too short for slow consumer recovery; disk full from unbounded topics; no tiered storage plan.

Checklist:

Classify topics by replay and compliance needs.
Monitor disk per broker and topic size.
Align retention with consumer lag SLAs.
Use tiered storage or export to lake for archives.