I'm using Kibana 7.5.2 and Elastic search 7.
My First approach: I'm sending log data and system data using fluentd and metricbeat respectively to my Kibana server. Everything working fine. I'm able to see data on the discovery page. But I had a large amount of data. so I added Kafka in between servers.
My second approach: Now I'm sending log data and system data to Kafka. and then from Kafka, I'm sending it to the Kibana server. For system data via metricbeat, I'm getting @timestamp field in Kibana, and for log data via fluent, I'm not getting @timestamp field.
If I'm running Kafka server individually for both one by one, everything works fine. but if I run both of them together. System data is not showing in the discovery tab.
Kibana index for system data: metricbeat-*
Kibana index for log data: prelive-data*
fluentd conf
<match laravel.**>
@type kafka2
# list of seed brokers
brokers <my-Broker-ip>:9092
<buffer topic>
@type file
path /var/log/td-agent/buffer/td
flush_interval 3s
</buffer>
# topic settings
# data type settings
<format>
@type json
</format>
# topic settings
topic_key prelive-data-log
default_topic prelive-data-log
# producer settings
compression_codec gzip
# max_send_retries 1
required_acks -1
</match>
metricbeat.yml
#----------------------------- KAFKA --------------------------------
output.kafka:
# # initial brokers for reading cluster metadata
hosts: ["<my-broker-ip>:9092"]
topic: 'metricbeat-7.6.0-2020.03.25-000001'
worker.properties of Kafka server for system data (metricbeat)
offset.storage.file.filename=/tmp/connectm.offsets
bootstrap.servers=:9092
offset.flush.interval.ms=10000
rest.port=10084
rest.host.name=localhost
rest.advertised.port=10085
rest.advertised.host.name=localhost
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
plugin.path=/usr/share/java
filesource.properties of Kafka server for system data (metricbeat)
name=config-name
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=1
topics=metricbeat-7.6.0-2020.03.25-000001
key.ignore=true
schema.ignore=true
connection.url=http://<ES-IP>:9200
connection.username=username
connection.password=password
type.name=kafka-connect
~
worker.properties of Kafka server for system data (fluentd)
offset.storage.file.filename=/tmp/connectf.offsets
bootstrap.servers=:9092
offset.flush.interval.ms=10000
rest.port=10082
rest.host.name=localhost
rest.advertised.port=10083
rest.advertised.host.name=localhost
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=false
value.converter.schemas.enable=false
internal.key.converter.schemas.enable=false
internal.value.converter.schemas.enable=false
plugin.path=/usr/share/java
filesource.properties of kafka server for system data (fluentd)
name=<config-name>
connector.class=io.confluent.connect.elasticsearch.ElasticsearchSinkConnector
tasks.max=2
topics=prelive-data-log
key.ignore=true
schema.ignore=true
connection.url=http://<my-ES-ip>:9200
connection.username=username
connection.password=password
type.name=kafka-connect
I'm running my Kafka server /usr/bin/connect-standalone worker.properties filesource.properties
I have been stuck here for a week. Any help would be appreciated.