Airflow - no module named "airflow.providers" or "airflow.contrib.providers"

user_5573294 0
Viewed 5652

We are trying to import both MongoHook and GCSToLocalFilesystemOperator into our Airflow Project:

...per this post, "Providers package is no longer included with Airflow, but you can separately install them with pip using the specific backport package"

I have had no luck with this. For our DockerFile, we started using this image from the puckel/docker-airflow repo on GitHub.

I've tried adding these two lines to our requirements.txt

apache-airflow-backport-providers-mongo
apache-airflow-backport-providers-google

...and I've tried adding these lines to our DockerFile:

    && pip install apache-airflow-backport-providers-mongo \
    && pip install apache-airflow-backport-providers-google \

...and we're getting the errors mentioned in the title. How can I install these into our docker image/container to use them in our Airflow project?

edit: checking quickly to see if the issue was that I simply wasn't rebuilding the image

edit2: rebuilding the image was needed; however, now we are receiving the error No module named 'airflow.utils.process_utils', which we get when importing GCSToLocalFilesystemOperator. Not sure why Airflow took providers out of the base install, this is very frustrating...

airflow docker python
1 Answers

The answer to my original question was that I needed to rebuild the docker image (DOH).

The answer to my second question, I think, was that I was installing these packages twice (once with an explicit pip install in the Dockerfile, and once in the requirements.txt, which is also pip installed in the Dockerfile). When I removed the duplicate install, it seems to be working now.

user_55732940
Related