I've got a Kafka topic (version 1.0.0) with time-series data. I want to find all records with a specific key and where the record timestamp is within a given time interval.
My current approach is as follows:
- Create a KafkaConsumer with
enable.auto.commitset to false (thus avoiding committing offset when polling) - Find the partition that holds the key I'm looking for
- Assign that partition to the KafkaConsumer
- Through
offsetsForTimes, find the offset closest to the start of the time interval seekthe KafkaConsumer to start reading at that offset- Do
pollin a loop, accumulating all records with matching key until a record is returned with a timestamp later than the end of the time interval
This kind of works. Here are some issues:
- the
pollfunction sometimes returns empty lists before the end is reached. - if the end of the interval is right now, the polling won't stop until a record in the future is written to the topic.
- is it safe to reuse the same consumer group id on multiple threads?
Is there a better way to query for historical records?