CMPE
0
How can I list all the topics that have data flowing in them?
I have a Kafka cluster (Confluent 7.2.1) with zookeeper. And I have over 120 topics where some of them are not active. I can list all the topics using kafka-topics --list --bootstrap-server.... This will return the list of all the topics.
I am looking to list the topics that currently have data in it. I have data on my topics for up to 7 days.
Thank you
Out of the box, there is no easy way to do this.
You would need to loop over every topic, then pass name to GetOffsetShell command with -1 and -2 flags to get start and end offsets, then compare them; when they match, the topic is currently empty, but that doesn't mean that a new producer didn't spin up within the few seconds you run the tool.
Alternatively, there may be metrics tools like Prometheus exporters that can query the disk-usage of the broker log directories and report empty folders