Apache Flink: aligning watermark among parallel tasks

user_2257122 0
Viewed 982

We are using periodic event time window with watermark. We have currently 4 parallel tasks in our Flink App.

During the streaming process, all the 4 tasks' watermark values must be close to trigger window event.

For example;

Task 1 watermark value = 8

Task 2 watermark value = 1

Task 3 watermark value = 8

Task 4 watermark value = 8

Task 2 is waiting for log to update its watermark. However, the condition can occur before Task 2's update and we want to fire the window event before it.

Is there any mechanism to align all the parallel tasks' watermarks or fire the window event without waiting for other tasks?

apache-flink streaming java
1 Answers

This was already answered in the comments by @Til Rohrmann, main answer:

If you want to look for 3 senders which are the same person, then you should keyBy the person and then simply use a count window to generate the alerts. Maybe you also wanna filter out duplicate senders for the same person.

Followup question:

But we need to consider also the event time. For example 3 senders in 10 minutes(event time not process time)

Followup answer:

Then you can use an event time window with a custom trigger which fires when it has seen 3 elements.

I suppose the critical conclusion is: If you want to trigger a count of something, keyBy that field.

user_9837220
Related