How to query Kafka topic for records in a partition with timestamp inside interval?

Viewed 666

I've got a Kafka topic (version 1.0.0) with time-series data. I want to find all records with a specific key and where the record timestamp is within a given time interval.

My current approach is as follows:

  1. Create a KafkaConsumer with enable.auto.commit set to false (thus avoiding committing offset when polling)
  2. Find the partition that holds the key I'm looking for
  3. Assign that partition to the KafkaConsumer
  4. Through offsetsForTimes, find the offset closest to the start of the time interval
  5. seek the KafkaConsumer to start reading at that offset
  6. Do poll in a loop, accumulating all records with matching key until a record is returned with a timestamp later than the end of the time interval

This kind of works. Here are some issues:

  • the poll function sometimes returns empty lists before the end is reached.
  • if the end of the interval is right now, the polling won't stop until a record in the future is written to the topic.
  • is it safe to reuse the same consumer group id on multiple threads?

Is there a better way to query for historical records?

0 Answers
Related