How to consume messages from the beginning to a particular offset

user_1879605 0
Viewed 853

I have the map between partitions number and offsets number of a given topic in kafka. I would like to consume all messages starting from the beginning to those particular partition/offset mapping using golang. In a nutshell I would like to know whether it is possible to consume messages in kafka from the beginning to a particular offset.

I would like to know whether there are built-in tools or libraries that can help me do before I start implementing something.

sarama confluent-platform apache-kafka go
1 Answers

I am posting this as an answer because I am like 99.9% convinced that this is correct - you cannot specify the end offset on the Consumer. The Consumer API supports OOTB:

  1. Start at the beginning.
  2. Start at any arbitrary position.
  3. Start at the end.

Note that there's no option for specifying ranges but you can easily do that yourself: simply start at the beginning and stop when you have read whatever number of messages you need (e.g. your final offset).

See the section Controlling Consumer Position in the Official API

user_3454900
Related