How do you find what topics have data in Apache Kafka?

user_19548965 0
Viewed 32

I recently needed to know which topics have data in our Apache Kafka instance. I don't really care how many messages are there or what's in them, etc. I just need to know which topics are populated.

apache-kafka
1 Answers

Pseudocode: Get all topics from Kafka, loop over them and sum kafka-get-offsets.

import subprocess
topic_lines = subprocess.check_output(['kafka-topics', '--bootstrap-server', 'admin:9092', '--list']).decode('utf-8')
for topic in topic_lines.rstrip().split('\n'):
    offset_lines = subprocess.check_output(['kafka-get-offsets', '--bootstrap-server', 'admin:9092', '--topic', topic]).decode('utf-8')
    messages = 0
    for line in offset_lines.rstrip().split('\n'):
        messages += int(line.split(':')[2])
    if messages > 0:
        print(topic,messages)

Output is each topic and how many messages exist in that topic, if there is at least 1 message.

user_195489650
Related