We're running DBT in Airflow on a GCP Compute Engine using Docker and docker-compose. In the docker-compose.yml file for our Airflow deployment, the DBT repo is attached as a volume:
webserver:
...
volumes:
- ../our-dbt-repo:/usr/local/airflow/dbt
Running dbt-run usually generates a /logs directory with DBT logs. However, running dbt-run from the docker container on the GCP machine is throwing the error [Errno 13] Permission denied: 'logs'.
To test permissions, I ran docker exec -it <DockerContainerID> bash from the command line of the GCP machine (while the docker container is running) to get into the running docker container, and ran cd /usr/local/airflow/dbt/ && touch file.txt, and received the error: touch: cannot touch 'file.txt': Permission denied. So it seems clear that no files can be added to the /dbt folder that was added as a volume in the Docker Container, which is why the logs cannot get written.
Is there a way to give our /dbt volume permissions to write logs? Perhaps we can write DBT logs somewhere else (not in the container on the GCP server), that way there are no writes required in the /dbt volume on the container?