How to handle Timeout exceptions during Offset Commit in a Kafka Streams application

user_6596263 0
Viewed 1969

If there is a timeout exception during committing offsets in a Kafka Streams application, does the ProductionExceptionHandler come into play?

Timeout of 60000ms expired before successfully committing offsets

I'm using a Spring cloud stream based Kafka streams application and I do have a ProductionExceptionHandler implementation configured to LogAndContinue. But it seems like the control doesn't come to the exception handler for this timeout exception and all stream threads die anyway.

What's the correct way to handle this? Here is the full error.

[-StreamThread-1] o.a.k.s.p.i.AssignedStreamsTasks         : stream-thread [stream-app-0d77ace1-4d20-413b-a71a-94cca3e620e0-StreamThread-1] Failed to commit stream task 0_3 due to the following error:
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets {example-topic-3=OffsetAndMetadata{offset=139191, leaderEpoch=null, metadata='AQAAAXU0ut1W'}}
2020-10-17 04:05:23.410 ERROR [ratelimit-transformer,,,] 19 --- [-StreamThread-1] o.a.k.s.p.internals.StreamThread         : stream-thread [stream-app-0d77ace1-4d20-413b-a71a-94cca3e620e0-StreamThread-1] Encountered the following unexpected Kafka exception during processing, this usually indicate Streams internal errors:
org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets {example-topic-3=OffsetAndMetadata{offset=139191, leaderEpoch=null, metadata='AQAAAXU0ut1W'}}
2020-10-17 04:05:24.149 ERROR [ratelimit-transformer,,,] 19 --- [-StreamThread-1] org.apache.kafka.streams.KafkaStreams    : stream-client [stream-app-0d77ace1-4d20-413b-a71a-94cca3e620e0] All stream threads have died. The instance will be in error state and should be closed.
Exception in thread "stream-app-0d77ace1-4d20-413b-a71a-94cca3e620e0-StreamThread-1" org.apache.kafka.common.errors.TimeoutException: Timeout of 60000ms expired before successfully committing offsets {example-topic-3=OffsetAndMetadata{offset=139191, leaderEpoch=null, metadata='AQAAAXU0ut1W'}}
apache-kafka-streams spring-cloud-stream java
1 Answers

Currently TimeoutException is not very well managed by Kafka Streams and will eventually cause your StreamThread to crash. Most of a the time a timeouts happen due to network issue or kafka unavailability.

The error message logged by Kafka Streams is explicit "All stream threads have died. The instance will be in error state and should be closed."; So, you can stop and restart your KafkaStreams application but if it's a network issue restartingyour app will not help.

Note that there is some works in progress :

You can configure an java.lang.Thread.UncaughtExceptionHandler on your KafkaStreams instance to handle that exception (KafkaStreams JavaDoc)

user_36608590
Related