How to visualize message from Kafka in Kibana

user_10504469 0
Viewed 2413

I am gathering data with Packetbeat, Metricbeat, and Filebeat and send them to Logstash through Kafka and then Elasticsearch, finally to Kibana.

When I didn't use Kafka and Logstash. I could easily visualize in Kibana through default dashboard setup and many available fields. I can create dashboards showing CPU, RAM, disk space metric data.

However, when I use Kafka and Logstash, all the data is stored in message field. I don't have many fields which I can use to visualize it in dashboards. I think Kafka puts everything in the message. I want to visualize all the system metrics data and others.

enter image description here

enter image description here

How can I utilize data in the message field to visualize it?

The following link appears as a similar question, but my situation is a little bit different, and it does not solve my case. SO similar question

My Logstash configuration : 

input {
    kafka {
            bootstrap_servers => "localhost:9092"
            topics => ["testkafka"]
            codec => json    <=  Val's answer right here.
    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => ["testkafka"]
    codec => json
elastic-stack logstash kibana elasticsearch apache-kafka
1 Answers

The solution is to make sure that Logstash parses the messages it gets from Kafka with the JSON codec

kafka {
        bootstrap_servers => "localhost:9092"
        topics => ["testkafka"]
        codec => json                              <-- add this line
}
user_46045790
Related