Flink Job Fail on Kafka Data peaks

user_9645434 0
Viewed 36

I have a flink Job that's read Data from one Kafka topic enrich the data and send it to output Kafka topic.

From my source kafka I get consistent 3mbs and 200 messages per second. But sometimes I can get peaks of 70mbs and 9k message per seconds.

When peaks like that's happened my output topic quota can't handle the amount of messages I send. So I get busy 100% on my sink that cause backpresure causing failed checkpoints thats cause Job failure. And there times that the job won't restart successfuly.

Is there a way that flink handle peaks of data like that without causing job restart. (Like slowing down the read from the source topic)

apache-flink apache-kafka
1 Answers

Flink automatically forwards backpressure to the sources, and slows down reading to the capacity of the pipeline. The problem isn't that the sources aren't throttled, but that during severe backpressure the checkpoint barriers aren't making sufficiently rapid progress.

Unaligned checkpoints and buffer debloating are both designed to help in situations like this. See the docs on checkpointing under backpressure for details. (Although unaligned checkpoints are available since Flink 1.11, this feature works best since Flink 1.13. Buffer debloating is available since 1.14.)

You could also consider increasing the checkpoint timeout, and you might want to investigate autoscaling as a way to scale up the cluster during times of peak load.

user_20008230
Related