Airflow Relative Importing Outside /dag Directory

user_1881080 0
Viewed 13658

I haven't been able to move common code outside of the dag directory that airflow uses. I've looked in the airflow source and found imp.load_source.

Is it possible to use imp.load_source to load modules that exist outside of the dag directory? In the example below this would be importing either foo or bar from the common directory.

── airflow_home
     |──── dags
     │   ├── dag_1.py
     │   └── dag_2.py
     ├── common
         ├── foo.py
         └── bar.py
airflow python
4 Answers

Add your airflow home path to PYTHONPATH

export AIRFLOW_HOME=/usr/local/airflow
export PYTHONPATH="${PYTHONPATH}:${AIRFLOW_HOME}"

Dockerfile 

ENV AIRFLOW_HOME=/usr/local/airflow
ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"
user_27385650

Other way instead of adding __init__.py file is to add the following include at the top of dag script:

import sys
import os
sys.path.insert(0,os.path.abspath(os.path.dirname(__file__)))
user_53981570

After running into the same problem, this fixed it for me:

import sys, os sys.path.insert(0,os.path.abspath(os.path.join(os.path.dirname(file),os.path.pardir))) from common.foo import *

user_61835020
Related