it seems that we are having same issue as described here: https://issues.apache.org/jira/browse/SPARK-20780
I am already aware that its a Kafka issue rather then spark, but still would like to get some advice on how to act until that issue would be resolved by the Kafka community.
Increasing the request.timeout is not much helping, because then we could get large queue. For example if the micro-batch is 10 seconds and the Kafka request.timeout.ms is 20 seconds, each time that the issue occurs, it would create a delay of around 20 seconds which would lead to 2 micro-batches queuing up. Obviously the more it happens, the more delays it causes which eventually leads to quite a large queue.
Any best practices / workarounds / tips for how to overcome that issue until it would be resolved ?