Kafka get timestamp based on offset and partition for monitoring time lag

user_6716114 0
Viewed 98

I have code in place to find offsets and TopicPartition from the KafkaConsumer, but can't find a way to just retrieve the timestamp based on that information.

I have looked through ConsumerRecord but since this is a monitoring service I do not think I should .poll() as I might cause some records to fall through if my monitoring service is directly polling from Kafka.

I know there's CLI kafka-console-consumer which can fetch the timestamp of a message based on partition and offset, but not sure if that's an SDK available for that.

Does anyone have any insights or readings I can go through to try to get time lag? I have been trying to find an SDK or any type of API that can do this.

kafka-consumer-api apache-kafka
2 Answers

There is no other way (as of 3.1) to do this - you can do consumer.poll. Of course, if you want to access only one, then you should set the max received records property to 1, so you don't waste effort. A consumer can be basically treated as an accessor to remote record-array, what you are doing is just accessing record[offset], and getting this record's ts.

So to sum it up:

  • get timestamp out of offset -> seek + poll 1,
  • get offset out of timestamp -> offsetsForTimes.
user_62046120

If I understood your question, given a ConsumerRecord, from Kafka 0.11+, all records have a .timestamp() method.

Alternatively, given a topic, (list of) partition(s), and offset(s), then you'd need to seek a consumer, with max.poll.records=1, then extract the timestamps from each polled partition after the seeked position.

The Confluent Monitoring Interceptors already do something very similiar to what you're asking, but for Control Center.

user_23086830
Related