How to shift from one consumer group to another consumer group by picking up last read message of previous consumer group

user_17953793 0
Viewed 46

I have a scenario where we have one consumer group and multiple topics in it, with each topic having multiple partitions.

Consumer C1, Topic : T1 , T2, T3

Since this is causing issue at time of rebalancing, we want to change this from single consumer to multiple consumer

Consumer C2 , Topic :T1
Consumer C3 , Topic :T2
Consumer C4 , Topic :T3

If I make this transition then C2 will start reading messages from beginning or latest, but what i want is, C2 should start reading from where the C1 has stopped processing.

kafka-consumer-api apache-kafka
2 Answers

Unfortunately, Kafka has no provided tool to do such a thing. But the APIs all exist for it.

You'd need to manually describe your existing consumer group (kafka-consumer-groups --describe), then parse through that output for specific topic partitions to see what offsets have been committed. (This script is a wrapper around a Scala class, if you don't want to parse CLI output and can import it instead)

Then you will need to start you own KafkaConsumer instance in code, set the new group.id and call seek for each topic partition. After seeked, you can call commitSync to store those offsets for the new group.

Afterwards, any new consumer for that group will start from those committed offsets.

The unused topic information in the original group will eventually be cleared out due to retention settings.

user_23086830

If C2, C3, & C4 use same consumer group as C1, new consumer will start processing where C1 stopped processing message. No additional configuration require. auto.offset.reset(earliest/latest) is used only when there is no information of Consumer group.

Since Re-Balancer will try to re assign partitions, this may cause high cpu for while, also vary based on data present on topics.

user_49582330
Related