Kafka: change in consumers number in a group

user_2444661 0
Viewed 17

I understand that Kafka semantics is that a consumer group must read a record only once. To achieve this, Kafka consumers maintain an offset, which is then conveyed to brokers with read requests so that brokers can send data accordingly to ensure that already read data is not resend(). But how does broker and consumers react when their is a change in consumer group, like addition of a new consumer or an existing consumer going down?

There are few things which needs to be considered here.

  • A consumer goes down, then how is its offset information taken into account while assigning its partitions to active consumers?
  • A new consumer joins, then how does system ensures that it doesn't read a data its consumer group has already read?
apache-kafka
1 Answers

If consumers join/leave a group, there's a consumer group rebalance. All consumers in the group will temporarily be suspended, then new partitions will be assigned to consume from.

If those consumers were processing, then there's a good chance that they'll re-consume the same data.

If you use transactions, the chance that happens could be a reduced as records will be consumed "exactly once". But this doesn't necessarily mean "successfully processed and offset committed" exactly once.

user_23086830
Related