Airflow - dags not beeing run

user_8579931 0
Viewed 680

python: 27

I have just installed airflow on a new vm GCP with a mysql backend. (sudo pip install apache-airflow and sudo pip install apache-airflow[mysql])

I have not created any dags of my own, but trying to run the examples to learn how it works.

I have altered the config to use 

executor = LocalExecutor

and done

export AIRFLOW_HOME=`pwd`/airflow  (in both shells)
airflow initdb

Webserver is started as "airflow webserver" and in a different shell i have started "airflow scheduler".

I have tried going into the web ui to stat task by clicking "DAGs", and then the run button. (where a dialoge opens asking if i want to run now)

The ones I've tried starting is "example_bash_operator, example_http_operator, example_python_operator", they now show as running. But the state does not change. I have also clicked so the 3 DAGs shows as "on".

When i go to browse -> Task instances and look the first ones that are supposed to start it shows:

All dependencies are met but the task instance is not running. 
In most cases this just means that the task will probably be scheduled soon unless:
.......

My servers doesnt give any meaningful output. My scheduler is just outputting

[2017-11-20 13:42:37,460] {jobs.py:1407} INFO - Heartbeating the process manager
[2017-11-20 13:42:37,460] {jobs.py:1443} INFO - Heartbeating the executor
apache-airflow google-cloud-platform python
1 Answers

I know it is a bit late but I also faced the same issue and so I want there to an answer for other people facing this issue.

So the issue is because airflow's scheduler looks for dags to run in the folder specified by the variable dags_folder in airflow.cfg. The example dags are stored where the airflow library is installed and that is the reason why the scheduler is not picking up anything to run. To fix it just change the value for dags_folder in airlfow.cfg to point to the folder containing the example dags. To find the example dag folder run find | grep 'example_bash_operator' from the root directory

user_34958140
Related