Retain kafka message in buffer

user_12420837 0
Viewed 421

I wish to write a producer that can retain the messages in Kafka per 16 seconds and after flush all messages to topic. Is possible do it? I am using the Kafka python.

The idea is: I have 100 messages, in normal behavior the producer will do:

Producer: message 1 message 2 message 3 ... message 100

Consumer: message 1 message 2 message 3 ... message 100

I want do it: Producer: message 1 message 2 message 3 ... message 100

Consumer: message 1 message 2 message 3...

[after 16s]

...message N-1 message N

[after 16s]

message 100

Is possible do it only use Kafka? Tkz

apache-kafka python
2 Answers

A Kafka topic already is somewhat of a buffer, and producers send messages in batches as well, not one by one.

You can increase size of that batch, depending on the kafka library you're using, and manually flush it, but it appears you want the consumer to be halted for a period of time, which can be done, but it'll also cause the consumer group to rebalance as it expects a continuous heartbeat from active consumers (another timeout setting that can be modified)

Other solution would be use a dequeue in your consumer code with a timer that is continuously added to, but only emptied on a schedule

And if you want messages N-1 and N, then resume back to some earlier event, you need to seek the consumer group to (near) end of the topic

user_23086830

You can pause your consumer(partitions) at the required time and continue to keep polling(or the broker with kick your consumer out as @OneCricketeer suggested) during which it will return no records on the paused partitions and then resume the consumer(partitions) after the time interval.

user_38858190
Related