Check message has been written to kafka topic using command line

user_11951828 0
Viewed 5166

Firstly, please note that using the java consumer API is not an option. Why it is not an option I am unable to disclose, but I must be able to do the following using a shell command.

I have a topic that I have written a message to, and I can confirm this is the case if I run ./kafka-console-consumer.sh with the --from-beginning option, but since this starts a consumer then the command gets stuck and requires manual intervention with a SIGINT. I have come close using --timeout-ms, however this is not ideal as unless I select a high value there is the possibility that the dump of the data is unreliable.
I would like to dump the output of console-consumer in such a manner that it can be grepped, or a suitable alternative method.

apache-kafka
1 Answers

When you write to Kafka, you can set in the producer acks which is the level of guarantee you want from the broker that the message has been received and written by the local broker and/or all replicas.

If you use this then you have no need to try and consume from the topic to determine if the record was written or not. This sounds like a really bad idea to try and do.

If you absolutely must use a command-line tool to do this (which, is not a good idea) then use kafkacat which can consume from any offset for any number of messages, e.g.:

  • Consume (-C) five messages (-c 5) from the beginning (-o beginning), or exit (-e) when end of partition is reached

    kafkacat -b localhost:9092 -t mytopic -o beginning -e -C -c 5

  • Consume (-C) ten messages (-c 10) from the end (-o -10), or exit (-e) when end of partition is reached

    kafkacat -b localhost:9092 -t mytopic -o -10 -e -C -c 10

  • Consume (-C) one messages (-c 1) at offset 42 (-o 42), or exit (-e) when end of partition is reached

    kafkacat -b localhost:9092 -t mytopic -o 42 -e -C -c 1
user_3506130
Related