I have created simple DAG using Bash Operator and BigQuery operator, there are approx 40 BigQuery tasks, I want to execute all 40 BigQuery task to execute parallel, only 10-15 tasks are executing parallel, I have tried with different Configuration parameters.
I also also need to execute 15-20 same kind of DAG, I need to execute all DAGs to execute parallel, all tasks from all DAGs.
My DAG is like
start >> bigquery task_1 >> end
start >> bigquery task_2 >> end
... ... ... ... ... ... ...
... ... ... ... ... ... ...
start >> bigquery task_n.. >> end
start and end is bash operator
My composer configuration as below:
Image version: composer-1.13.3-airflow-1.10.12 Python version: 3 Worker nodes" Node count: 3 Disk size (GB): 50 Machine type: n1-standard-2
I have tried with default Airflow configuration and well as below custom configuration and don't see much improvement.
custom configuration 1
parallelism = 300
dag_concurrency = 60
max_active_runs_per_dag = 15
enable_xcom_pickling = False
sql_alchemy_pool_recycle = 570
store_serialized_dags = False
min_serialized_dag_update_interval = 30
dag_concurrencymax_active_runs_per_dag = 60
custom configuration 2
parallelism = 230
dag_concurrency = 46
max_active_runs_per_dag = 15
enable_xcom_pickling = False
sql_alchemy_pool_recycle = 570
store_serialized_dags = False
min_serialized_dag_update_interval = 30
dag_concurrencymax_active_runs_per_dag = 46
Any suggestion will be highly appreciated.