Thread Sleep in the Kafka Listener

user_8903088 0
Viewed 415

I am trying to pause/resume the Kafka container. Using the following code snippet to do so:

kafkaListenerEndpointRegistry.getListenerContainer("MAIN").pause();

When I call the pause, I also need to do a thread.sleep so that messages in the batch are not processed. For every message in the batch, I am calling another API which has a rate limit. To maintain this rate limit, I need to stop the processing for the message.

If the Main thread sleeps, will it stop Listener from sending the hearbeat? Does it also stop the heartbeat thread in the background? Documentation says , "When a container is paused, it continues to poll() the consumer, avoiding a rebalance if group management is being used, but it does not retrieve any records. " But I am pausing the container and making the thread sleep. How will this impact the flow?

spring-kafka apache-kafka multithreading
1 Answers

You must never sleep the consumer thread, to avoid rebalancing.

Instead, reduce the max.poll.records so the pause will take effect more quickly (the consumer won't actually pause until the records received by the previous poll are processed).

You can throw an exception after pausing the consumer, but you will need to resume the container somehow.

I opened a new issue to improve this behavior https://github.com/spring-projects/spring-kafka/issues/2280

If you are subject to rate limits, consider using KafkaTemplate.receive() methods, on a schedule, or a polled Spring Integration adapter, instead of using a message-driven approach.

user_12407630
Related