I'm trying to figure out how to implement a workflow so that a sensor task wait for external dag to complete, for only wait for a certain number of days. It's a daily job so I'd like a sensor job to wait for example 3 days, and on the forth day send out an email, and either waiting or do some other task.
Could someone please help to shed some light on how to achieve this? Also how do we communicate the "days counter" from one day to another? Many thanks for your help.
You can use a ExternalTaskSensor with the following configurations:
timeout = 3 * 24 * 60 * 60 - 3 days in seconds, after that time sensor will failpoke_interval = 12 * 60 * 60 - 12h between sensor checks, you can adjust it to let say a check every hour. It will reduce number of times when you check the external dag statemode = "reschedule" - in this way the sensor will not occupy worker slot for 3 days, it will be scheduled, executed and if condition is not met it will be rescheduled to be executed in next poke_interval seconds. It's a good practice to use this mode for long running tasks.Additionally you can build your waiting DAG as wait_task >> [success_task , fail_task] where
wait_task is your sensorsuccess_task has trigger rule all_success and is followed when the sensors succeedsfail_task with all_failed trigger rule and handles scenario when sensor finally return false or timeouts