Airflow - Is it possible to run one day at a time(sequentially) using the backfill command?

user_2198720 0
Viewed 1550

Basically, I would like to run the backfill command for a whole month. But there are some tasks that depend on the data of the previous day. And as far as I know, this command runs each day at the same time.

airflow backfill adsquare_events -s 2017-01-01 -e 2017-01-31

Is there a way to make the backfill command to run one day at a time(sequentially)?

Cheers.

apache-airflow airflow-scheduler airflow
2 Answers

Put ExternalTaskSensor as a first step:

ExternalTaskSensor(
    task_id='wait_yesterday_run',
    external_dag_id='adsquare_events',
    execution_delta=timedelta(days=1),
    execution_date_fn=None,
    dag=dag,
)
user_76827230
Related