I currently have an Airflow instance running with docker-compose. In the future, I will be moving to a kubernetes cluster. So, Airflow will always be running in a docker container.
That being said, how do I run a DockerOperator when Airflow itself is inside a docker container?
It's a docker-in-docker inception problem that I don't fully understand how to mitigate.
Thanks!
You could map docker socket inside the container (/var/run/docker.sock) configure docker engine URL to connect to external Docker engine (Docker architecture is done in the way that you actually run client locally and the engine running your docker container runs elsewhere.
Just look for Docker-in-Docker term.
There is currently (as of 2.0.0 Docker Provider) a bug preventing doing that, but it is being addressed in https://github.com/apache/airflow/pull/16932 and it will be out in < week - or you can downgrade to previous version of the provider.
