Schedule A bi-weekly job to run at 5 PM on Friday

user_7517479 0
Viewed 368

I saw 2 discussions on bi-weekly job. However, both don't have a precise solution to this technical requirement. That's why, I am starting this thread.

I have a requirement to run a DAG every alternate Friday at 5 PM. So, if I start my DAG on 19 Mar, 2021, then it should run on the 19th of Mar, 2021 and the subsequent runs should happen on alternate Fridays. This would mean that after 19th Mar, 2021 the next run should happen on 02 Apr, 2021. How can I achieve it?

Thanks in advance!

airflow
3 Answers

After lot of deliberations and help from many in the community, I got the answer from AirFlow slack community.

def _skipcondition(execution_date, **_):
    first_valid_friday = datetime(2021, 4, 9)
    return (execution_date - first_valid_friday).days % 14 == 0
with DAG(dag_id='fortnightSchedule',default_args = default_args, schedule_interval = "0 17 * * 5", catchup=False) as dag:
    check_valid_friday = ShortCircuitOperator(
        task_id="check_if_correct_friday",
        python_callable=_skipcondition,
        provide_context=True,
        )
    rest_of_tasks = DummyOperator(
        task_id="dependent_dummy",
    )
check_valid_friday >> rest_of_tasks

Its a simple function that returns True when the number of days between execution date and first execution date is divisible by 14. Thanks for your help.

user_75174790

I do not think there is any straightforward way to do this, but I can provide some valid approaches by which you can do it.

Option 1

  1. Set the schedule to 0 17 * * 5 - which translates to At 17:00 on Friday.
  2. Use a BranchPythonOperator to get the last dag successful branch run date from airflow variables and check if it is 14 days from the current run date.
  3. The date will be stored as an airflow variable and updated only when the true branch (alternate Friday runs) succeed.
  4. If yes, continue the rest of the DAG, else, skip the rest of the tasks. An example of how to implement this is given in the above-mentioned link.

Option 2

  1. Set the schedule to something like 0 17 1,14 * 5 - which translates to At 17:00 on day-of-month 1 and 14 and on Friday. This will run twice a month.

Note: Option 1 is preferred over Option 2 as it will help tackle the requirement as expected.

Hope these help!

user_71224380

I think you can use datetime.timedelta function. you can execute DAG after 14days based on start_date. using this you can execute DAG below day.

  • 2021-03-19
  • 2021-04-02
  • 2021-04-16
  • 2021-04-16 ....

this can be the example.

import datetime

dag = Dag(
    dag_id="bi_weekly_dag",
    schedule_interval=datetime.timedelta(days=14)
    start_date=datetime.datetime(year=2021, month=4, day=19)
)
user_88542510
Related