Consuming exact number of events from Kafka

user_5605829 0
Viewed 298

Naturally, streaming apps are unbounded, but I have a new use case where I need to consume an exact number of messages or less (configurable, for example, 100 messages) from a Kafka topic. Then the app should stop.

The motivation is very simple, the flow is rarely used and no need for real-time, so there is no reason to have a permanent streaming app.

It is enough to invoke the app once in a while.

Is there a way to implement it using FlinkKafkaConsumer?

Adding a counter that will kill the app when it reaches the required number of messages is an option, but I prefer to use something more elegant.

flink-streaming apache-flink apache-kafka
2 Answers

You could create a wrapper SourceFunction for the FlinkKafkaConsumer, which delegates to it, and terminates when the target number of messages have been read. When all of the sources of a Flink streaming job are done, the workflow will automatically stop.

user_2317620

You can set max.poll.records to 100 and set adjust fetch.max.bytes to the number equal to the size of 100 messages. Eg: 1 mess = 10bytes => fetch.max.bytes = 100*10bytes

user_124553200
Related