We have an extraction job which executes Java code, that we encapsulated in a Dockerfile to keep our host machine clean.
Now we are considering introducing Airflow to automate this and other tasks . Our tasks run on different machines and should be scalable, so we want to execute everything inside docker containers. We use RabbitMQ for the queue and MySQL for the meta database.
Intuitively, I thought that installing a minimum Airflow python package dedicated to handling the CeleryExecutor part would be enough, but it seems, that any Airflow instance must hold a connection to the database, so it seems, that at the end I would have to use a similar (Docker) environment as the Airflow server/scheduler.
Now I was wondering if I have to inherit a base Airflow Image for every single job that I want to manage? In our case that would mean, that I have to get a little messy by installing the Java part on top of the inherited Airflow environment.
Do I really have to write custom Dockerfile parts for including Java or other environments where I would usually just inherit from a dedicated Docker image or is there another option that I dont see yet?
Starting docker inside docker is not an option in my opinion.