In our application, one DAG run processes 50records every 10mins. To load historical data(~80k) in a short period of time, we increased the max_active_runs to 3 and decreased the interval to 2mins.
When the DAG starts execution, in the first task, we pick up the first 50records that are eligible, mark them as IN-PROGRESS, and proceed.
The issue we noticed is, when multiple DAGs start their execution at the same time (when there is some delay from the previous DAG run), the same records are being picked up by more than one DAG run.
Is there a possibility to force a delay between multiple active DAG runs?