Airflow not loading dags in /usr/local/airflow/dags

Try airflow db init before listing the dags. This is because airflow list_dags lists down all the dags present in the database (And not in the folder you mentioned). Airflow initdb will create entry for these dags in the database.

Make sure you have environment variable AIRFLOW_HOME set to /usr/local/airflow. If this variable is not set, airflow looks for dags in the home airflow folder, which might not be existing in your case.

The example files are not in /usr/local/airflow/dags. You can simply mute them by edit airflow.cfg (usually in ~/airflow). set load_examples = False in 'core' section.

There are couple of errors may make your DAG not been listed in list_dags.

1. Your DAG file has syntax issue. To check this, just run python custom_example_bash_operator.py and see if any issue.
2. See if the folder is the default dag loading path. For a new bird, I suggest that just create a new .py file and copy the sample from here https://airflow.incubator.apache.org/tutorial.html then see if the testing dag shows up.
3. Make sure there is dag = DAG('dag_name', default_args=default_args) in the dag file.

It will be the case if airflow.cfg config is pointed to an incorrect path.

STEP 1: Go to {basepath}/src/config/

STEP 2: Open airflow.cfg file

STEP 3: Check the path it should point to the dags folder you have created

dags_folder = /usr/local/airflow/dags

I find that I have to restart the scheduler for the UI to pick up the new dags, When I make changes to a dag in my dags folder. I find that when I update the dags they appear in the list when I run airflow list_dags just not in the UI until I restart the scheduler.

First try running:

airflow scheduler

There can be two issues: 1. Check the Dag name given at the time of DAG object creation in the DAG python program

dag = DAG(
dag_id='Name_Of_Your_DAG', 
....)

Note that many of the times the name given may be the same as the already present name in the list of DAGs (since if you copied the DAG code). If this is not the case then 2. Check the path set to the DAG folder in Airflow's config file. You can create DAG file anywhere on your system but you need to set the path to that DAG folder/directory in Airflow's config file.

For example, I have created my DAG folder in the Home directory then I have to edit airflow.cfg file using the following commands in the terminal:

creating a DAG folder at home or root directory

$mkdir ~/DAG

Editing airflow.cfg present in the airflow directory where I have installed the airflow

 ~/$cd airflow
 ~/airflow$nano airflow.cfg

In this file change dags_folder path to DAG folder that we have created.

If you still facing the problem then reinstall the Airflow and refer this link for the installation of Apache Airflow.

You need to set airflow first and initialise the db

export AIRFLOW_HOME=/myfolder
mkdir /myfolder/dags
airflow db init

You need to create a user too

 airflow users create \
          --username admin \
          --firstname FIRST_NAME \
          --lastname LAST_NAME \
          --role Admin \
          --email admin@example.org

If you have done it correctly you should see airflow.cfg in your folder. There you will find dags_folder which shows the dags folder.

If you have saved your dag inside this folder you should see it in the dag lists

airflow dags list

, or using the UI with

airflow webserver --port 8080

Otherwise, run again airflow db init.

In my case, print(something) in dag file prevented printing dag list on command line.

Check if there is print line in your dag if above solutions are not working.