advanced

Kafka consumer groups

Use consumer groups so each partition is processed by one member per group while independent groups keep their own offsets and replay pace.

Consumer groups coordinate partition assignment: each partition consumed by one group member. Different groups reading the same topic maintain independent offsets — enabling pub/sub. Static membership and cooperative rebalancing reduce stop-the-world pauses.

Trade-off: max useful consumers per group bounded by partition count; more groups mean more offset storage and lag dashboards.

On interviews: add a new analytics pipeline without affecting billing consumer group offsets.

Common pitfalls: same group.id in staging and prod; consumers exceeding partitions sit idle; rebalance on every pod restart.

Checklist:

Unique group.id per application and environment.
Match consumer count to partitions intentionally.
Tune session timeout and max.poll.interval.
Dashboard lag per group and partition.