We have configured the Airflow and there are two airflow dags working fine. I have created a new dag and can see the same in the Airflow UI . But when I try to turn ON the Airflow Dag from UI it gives me the following error "DAG <sample_dag> seems to be missing".
I tried to troubleshoot the issue using the following ways:
- Firstly, I tried to run my pipeline py file to see if there are any issues with it :
python sample_dag_pipeline.py
['/home/airflow/airflow-install/dags/ABC_Middleware', '/home/airflow/airflow-install/dags', '/home/airflow/airflow-install/sample_folder/bin/python', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages', '/home/airflow/airflow-install/sample_folder/lib/python3.6/site-packages']
It seems to work fine.
- Then I try to run : airflow list_dags It gives me the following error:
(env) [airflow@Airflow-VM dags]$ airflow list_dags
[2021-02-02 08:41:34,503] {__init__.py:50} INFO - Using executor LocalExecutor
[2021-02-02 08:41:34,507] {dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow-install/dags
['/home/airflow/airflow-install/sample_folder/bin', '/home/airflow/airflow-install/dags/AirflowMiddleware', '/home/airflow/airflow-install/sample_folder/bin/python', '/usr/lib64/python36.zip', '/usr/lib64/python3.6', '/usr/lib64/python3.6/lib-dynload', '/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages', '/home/airflow/airflow-install/sample_folder/lib/python3.6/site-packages', '/home/airflow/airflow-install/dags', '/home/airflow/airflow-install/config', '/home/airflow/airflow-install/plugins']
[2021-02-02 08:41:35,044] {dagbag.py:259} ERROR - Failed to import: /home/airflow/airflow-install/dags/sample_dag_pipeline.py
Traceback (most recent call last):
File "/home/airflow/airflow-install/sample_folder/lib64/python3.6/site-packages/airflow/models/dagbag.py", line 256, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib64/python3.6/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 684, in _load
File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 678, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/home/airflow/airflow-install/dags/sample_dag_pipeline.py", line 8, in <module>
from abc.def.gef import sample_dag_jobs
ImportError: cannot import name 'sample_dag_jobs'
-------------------------------------------------------------------
DAGS
-------------------------------------------------------------------
ABC_V01
DEF_V01
Lemme show my sample_dag_pipeline.py file:
(env) [airflow@Airflow-VM dags]$ cat sample_dag_pipeline.py | head -20
import sys
import traceback
from importlib import reload
sys.path.insert(0, "/home/airflow/airflow-install/dags/ABC_Middleware")
from datetime import datetime
from abc.def.gef import sample_dag_jobs
dagutils = reload(cai_jobs_pipeline)
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.python_operator import PythonOperator
from airflow.utils.trigger_rule import TriggerRule
DAG_NAME = "ABC_DataPipeline_V1.0"
Observation:
So in the above sample_dag_pipeline.py file I am referring to the code present in other directory(ABC_Middleware) by first adding it in sys path and then importing the modules from it. I did the same for my other two dags(ABC_V01, DEF_V01) for them it is working fine but for this one it is not.
when I run the py file it prints me the sys paths and i can see the path added in the sys.('/home/airflow/airflow-install/dags/ABC_Middleware')
But when I run : airflow list_dags in the sys paths I am not able to see the sys path of my directory ('/home/airflow/airflow-install/dags/ABC_Middleware')
I have tried many options but all seems to get stuck here only. Can anyone suggest what could have happened?