I'm running a DAG test_dag.py which is structured in the following way in my Google Cloud Storage Bucket.
gcs-bucket/
dags/
test_dag.py
dependencies/
__init__.py
dependency_1.py
module1/
__init__.py
dependency_2.py
Airflow detects the DAG, test_dag.py, which tries to import from depencies/dependency_1.py, (which imports successfully) and dependencies/module1/dependency_2.py which gives the error Broken DAG: [/home/airflow/gcs/dags/test_dag.py] module 'dependencies' has no attribute 'module1'.
The line causing this is from dependencies.module1 import dependency_2.
This seems to indicate to me that Cloud Composer is unable to import from a subdirectory within dependencies/, and when I look at their dependencies documentation here, the example they give is only one directory level down from /dags (and is only 1 file rather than being a full python package).
Here is the weird part though -- it runs successfully when I run this locally in Airflow (not on Cloud Composer). So I'm at a loss for why my imports would work locally but not on Cloud Composer.
I've also tried importing everything from within my __init__.py files, which gives me the same attribute error, and moving my dependencies a level up into gcs-bucket/ where they can't seems to be found at all.
When I print out __file__ from with my DAG I get /home/airflow/gcs/dags/test_dag.py and when I print sys.path I get:
['/usr/local/bin', '/opt/python3.6/lib/python36.zip', '/opt/python3.6/lib/python3.6', '/opt/python3.6/lib/python3.6/lib-dynload', '/opt/python3.6/lib/python3.6/site-packages', '/usr/local/lib/airflow', '/home/airflow/gcs/dags', '/etc/airflow/config', '/home/airflow/gcs/plugins']
I'm totally at a loss here, any help would be much appreciated. Thank you.
EDIT: It seems that Cloud Composer does not like when dependencies try to import other dependencies (see comments below). Wondering if there is some way around this?