Is it possible to log all incoming messages in Apache Kafka

user_6162262 0
Viewed 3665

I need to know if it possible to configure logging for Apache Kafka broker to write all produced/consumed topics and it's messages. I've been looking the log4j.properties but none of the suggested properties seems to do what I need.

Thanks in advance.

Looking the generated logging files by Kafka, none of them seems to log the messages written in the different topics.

UPDATE: Not exactly what I was looking for, but for anyone looking something similar I found: https://github.com/kafka-lens/kafka-lens which provides a friendly GUI to view messages on different topics.

apache-kafka
1 Answers

I feel like there's some confusion with the word "log".

As you're talking about log4j, I assume you're talking about what I'd call "application logs". Kafka does not write the records it handles in application/log4j logs. In Kafka, log4j logs are only used to trace errors and give some context about the work brokers are doing.

On the other hand, Kafka write/read records into/from its "log", the Kafka log. These are stored in the path specified by log.dirs (/tmp/kafka-logs by default) and are not directly readable. You can use the DumpLogSegments tool to read these files if you want, for example:

bin/kafka-run-class.sh kafka.tools.DumpLogSegments \
    -f /tmp/kafka-logs/topic-0/00000000000000000000.log
user_17651890
Related