I suddenly got exceptions of type in production Kafka
ERROR[pool-XX-thread-YY] org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - [Consumer clientId=someclientid, groupId=somegroup] Offset commit failed on partition SomeTopic-SomePartition at offset SomeOffset: The request timed out.
This occurred for 3.5 seconds from a lot of different services (clients) (different threads\different topics\different partitions) and than just self healed... The offset commit configurations is 5 second auto-commit for all those clients.
Couldn't track anything from kafka broker logs except of some re-balancing right of one group (out of 10 had that issue) which is normal when heart-beating fails, in the metrics server I can see some spikes of commit latency which is the symptom I guess and some TCP spikes on 1 broker (out of 3)
How can I start investigate it? what can cause such an issue? where should I look when things like that is occurring?
Attaching photos of some graphs here:
TCP Spike in server-3
Commit latency spike
Group syncs
Heartbeats
