How to set time object on SLA in airflow instead of timedelta?

user_6645617 0
Viewed 352

I'm trying to implement SLA in my airflow DAG.

I know how SLAs work, you set a timedelta object and if the task does not get done in that duration, it will send an email and notifies that the task is not done yet.

I want some similar functionality, but instead of giving duration, I want to set specific time in SLA. For example, if the task is not done due to 8:00 AM, it sends the email and notifies the manager. Something like this:

'sla': time(hour=8, minute=0, second=0)

I have searched a lot, but nothing found.

Is there any solution for this specific problem? or any other solutions than SLA?

Thanks in advance.

airflow-scheduler airflow python-3.x python
1 Answers

SLA param of BaseOperator expects a datetime.timedelta object, so there is nothing more to do there. Take into consideration that SLA represents a time delta after the scheduled period is over. The example from the docs supposes a DAG scheduled daily:

For example if you set an SLA of 1 hour, the scheduler would send an email soon after 1:00AM on the 2016-01-02 if the 2016-01-01 instance has not succeeded yet.

The point is, it's always a time delta from the schedule period which is not what you are looking for.

So I think you should take another approach, like schedule your DAG whenever you need it, execute the tasks you want and then add a sensor operator to check if the condition you are looking for is met or not. There are a few types of sensors depending on the context you have you could choose from them.

Another option could be, create a new DAG dedicated to check if your tasks executed in the original DAG were successfully executed or not, and act accordingly (for example, send emails, etc.). To do this you could use an ExternalTaskSensor, check online for tutorials on how to implement it, although it may be simpler to avoid cross DAG dependencies as stated in the docs.

Hope that this could point you into the right direction.

user_105692200
Related