Earlier this year I took Data Engineering Zoomcamp and the Docker file which went with the docker-compose.yaml file is basically this:
FROM apache/airflow:2.2.3
ENV AIRFLOW_HOME=/opt/airflow
USER root
RUN apt-get update -qq && apt-get install vim -qqq
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
SHELL ["/bin/bash", "-o", "pipefail", "-e", "-u", "-x", "-c"]
WORKDIR $AIRFLOW_HOME
COPY scripts scripts
RUN chmod +x scripts
USER $AIRFLOW_UID
However, when I add another package (kaggle for example) to requirements.txt there is an error that it will be installed at /root/.local/bin. So I simply moved the pip install to after the
USER $AIRFLOW_UID
I can login to the worker container and import the package just fine after that. The original requirements.txt contains these lines.
apache-airflow-providers-google
pyarrow
To my surprise when these are installed like in the original Dockerfile (as root) I can connect to the worker container and import them. My question is - why are some packages installed by the root user available to the default user in the container while others are not? Does it have to do with how the python package you are installing is coded?