We have one airflow DAG which is accepting input from user and performing some task. we want to run same DAG simultaneous with different input from user. we found multiple links for simultaneous task run but not able to get info about simultaneous run. so if we triggered DAG with two diff inputs from cli then its running fine with two instances but just want to understand that both instances are running independently or its waiting for one instance to finish and then trigger the other ?
1 Answers
All of what you mention can be done. Tasks can be executed in parallel.
It's just the proper configuration of max_active_runs wait_for_downstream and depends_on_past
DAG parameters:
max_active_runs - maximum number of active DAG runs
if you just want the DAGs to be able to execute two jobs in parallel (with no conditions between two distinct runs) then set max_active_runs=2
Operators parameters:
wait_for_downstream - when set to true, an instance of task X will wait for tasks immediately downstream of the previous instance of task X to finish successfully or be skipped before it runs
depends_on_past - when set to true, task instances will run sequentially and only if the previous instance has succeeded or has been skipped.
Related