How to change offset of a topic during runtime?

Sai Satwik Kuppili 0
Viewed 28

I have a Producer for kafka topic which keeps on pushing some messages to kafka topic. And also I have another service reading these messages from topic.

I have an business use-case, where sometimes consumer need to ignore all the messages which are already there in queue and start processing only new upcoming messages. Can this be archived without stopping and restarting the kafka server.

I am working on GO. So if kafka supports such requirement, is there any way I can change configuration of consumer to start consuming from latest message using sarama GO client.

Thank you in advance.

sarama kafka-consumer-api apache-kafka go
1 Answers

You could use a random UUID for consumer group id, and/or disable auto commits, then you can start at the latest offset with

config := sarama.NewConfig()
config.Consumer.Offsets.Initial = sarama.OffsetOldest

(copied from Sarama example code)

Otherwise, Kafka consumer API should have a seekToEnd function, but it seems to be exposed in Sarama as getting high watermarks from consumer for every partition, then calling ResetOffets on a ConsumerGroup instance. Note: the group should be paused before doing that.

OneCricketeer0
Related