kafka consumer offset to earliest

kimo815 0
Viewed 34

I'm new to Kafka. currently, I have created a java consumer on a Kafka topic. What I want is to get only the new messages when I start my java application.

After checking on the internet I found that I can do that by setting the property: auto.offset.reset: earliest, but we don't want to use that. Also after doing some research I found that maybe we can do that by updating the consumer offset to the earliest one. Can anyone help to know how I can do that, please?

Note that I'm using KStream to subscribe to the topic. Kafka version 2.4.0 subscribe source code:

StreamsBuilder sb = new StreamsBuilder();
sb.stream("topic").foreach((t, u) -> {...})
apache-kafka-streams apache-kafka spring java
2 Answers

updating the consumer offset to the earliest one

That's the default behavior and configuration for auto.offset.reset when your application.id is brand new.

Note that i'm using KStream to subscribe the topic.

Then you can't seek at runtime (or at least, shouldn't, assuming you're using stateful processors)

Use the Application reset tool which helps manage all topics used within your topology.

OneCricketeer0

Can you specify why you don't want to use auto.offset.reset?

That would be the default approach to do what you want. Other than that, you'll need to go and get the latest offset for each partition of the topic and "point" your consumer to that offset on the partition you want to consume.

LeYAUable0
Related