I have a dag with the following parameters:
start_date=datetime(2020, 7, 6)
schedule_interval="0 12 * * *",
concurrency=2,
max_active_runs=6,
catchup=True
I had to re-process a year's historical data, so I did a reset of dag run status for past one year. In the mid of re-processing, I realised I need to re-process a few latest days first due to some business priority change, but airflow seems be a bit random in picking which days to run, though often favours old days more, so my tree view of the dag run is a bit messed up, and it's going to take quite a while to catch up all runs.
I had two choices:
- Set all old dag run days to failure
- Delete old dag run days
To avoid generating excessive number of failure notification, I chose the 2nd option. Here is a quick illustration. Before I delete, in my tree view, I have:
- 1 Sep 2021 - 1 Jan 2022: dag run successful
- 2 Jan 2022 - 3 Jan 2022: dag running
- 4 Jan 2022 - 1 Aug 2022: dag scheduled
- 2 Aug 2022 - 6 Aug 2022: dag run successful
- 7 Aug 2022 - 1 Sep 2022: dag scheduled
To speed up the process of August data, I deleted dag runs scheduled between 4 Jan and 1 Aug, now the tree view now becomes
- 1 Sep 2021 - 1 Jan 2022: dag run successful
- 2 Jan 2022 - 3 Jan 2022: dag running
- 2 Aug 2022 - 6 Aug 2022: dag run successful
- 7 Aug 2022 - 1 Sep 2022: dag scheduled
Note that dag runs between 4 Jan 2022 and 1 Aug 2022 are now completely gone from the tree view.
Unfortunately, because the latest dag run is 6 Aug 2022 and airflow thinks there is only runs starting from 7 Aug to catchup and all deleted runs between 4 Jan and 1 Aug are hence ignored.
So my question now is, if I don't want to re-process those days that have already been re-processed, is there a way for me to tell airflow I need to re-run those days I have deleted?