Can Kafka consumer be paused for definite interval and resume automatically after the interval

Viewed 816

Can kafka consumer be paused for a definite time interval say 2 mins and get resumed automatically once the given time interval is over. Or can we provide time window to pause command?

2 Answers

Yes, we are using this for very very long processing time and in order processing of events, there is api in kafka client called pause and resume, you need to remember that in order that your consumer won't die you must keep calling poll() , pause makes it that poll request won't return any new records but will refresh the "timers" on poll request, when you wish you can call the resume and then poll again,

I am not a fan of this way of using kafka, as I would like kafka to be more of real time streaming machine and it somehow felt to me like against that logic , but it is working,

Yes, KafkaConsumer has a pause method. As a parameter it takes a collection of partitions. This method does not affect partition subscription and does not cause a group rebalance. Later, you can use resume method to resume a subscription.

Related