I have logs coming into 1 topic of Kafka, I am parsing them and putting them into 2 separate Kafka topic(having 10 partitions each), thereafter I am putting them into elasticsearch via kafka-elastic sink using 10 tasks. But I am facing serious lag of about 10 minutes lag for 30 minutes of continuous logs which in-turn increases for more logs. Please suggest to narrow the gap.
Ps. I checked my parser works fine and puts the data into the 2 topics in no time. Also I am using latest elasticsearch 6 and kafka.