Airflow - Filling up DagBag is really slow

We are exploring using Apache Airflow to replace the current orchestrator tool for our ETL. Our current ETL over 600 different tasks with complex dependencies. These tasks are mostly calls to Informatica PowerCenter workflows but also handful of database procedures and shell scripts

In order to test the UI, I created a DAG with 600 tasks. Rather than using the actual custom operators I have created, I used the out of the box BashOperator to print date, the simplest command I can think of. Then I set the dependencies to match our current ETL. There are over 2000 dependency relationships but I limited to about 1400 dependency relationships for the test.

import airflow
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta


default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': airflow.utils.dates.days_ago(1),
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5)
}

dag = DAG(dag_id='Test_ETL',
          default_args=default_args,
          schedule_interval=timedelta(days=1))


Task1 = BashOperator(task_id="Task1",bash_command="date",dag=dag)
Task2 = BashOperator(task_id="Task2",bash_command="date",dag=dag)
.
.
Task600 = BashOperator(task_id="Task600",bash_command="date",dag=dag)

Task1 >> Task2
Task2 >> Task3
Task4 >> Task5
.
.
Task500 >> Task600

The problem is Airflow takes few minutes to do the "filling up the DagBag". And this happens all the time and in every step takes several minutes because of this. In fact, I had to increase the timeouts as the filling the DagBag would not complete.

I investigated dag_pickle (donot_pickle = false, scheduler in --do_pickle mode) and yet dag_pickle table (in MySQL) is empty.

It took over 8 hours to run date command 600 times in bash! The whole ETL takes 8 hours to extract and load nearly a billion rows every night.

Without making this work, it is impossible for us to recommend Airflow as a replacement (which is a shame).