Spring Boot Kafka consumer.. reads from kafka stream and ingests into Elastic Search (batch processing mode). kafka-clients:2.0.1 kafka_2.11 spring-kafka:2.7.7
2020-09-22 01:10:53 [kafka-coordinator-heartbeat-thread | elastic-consumer] ERROR o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=elastic-consumer-4, groupId=elastic-consumer] Offset commit failed on partition daily.metrics-24 at offset 204745488133: This is not the correct coordinator.
This issue is intermittent, Current consumption and ingestion rate is 800k per ms.
Tried to find a sweet spot and reduced the occurrence of this issue with below configuration. This process runs 24*7 and ingests billions of records. The error occurs for very few batches (about 1M rows). Any advice in completely solving this problem.
application.yml
auto-offset-reset: latest
enable-auto-commit: true
poll-timeout: 9000 (Set the max time to block in the consumer waiting for records)
concurrency: 5
max-partition-fetch-bytes: 20000000
max-record-per-poll: 35000