Airflow settings to run Celery workers in different Docker container?

user_5305401 0
Viewed 2345

I am running Airflow in docker container. I have created a separate container to run Postgres server and Rabbitmq server, connected these containers using docker network - by following this nice article. Now my Airflow docker container is running and connected to other containers using docker network - the process went smooth so far. The problem is how to run airflow webserver, airflow scheduler and airflow worker in the same container. After some research I found: it is recommended to run one service in one container. Now I have two solutions

  1. Run multiple services in the same Airflow container - which I could not figure out a simple way implement being a new bee in Docker.
  2. Create separate containers to run Celery worker and Airflow scheduler - but in the airflow.cfg file the setting related to Celery are: broker_url = 'amqp://guest:guest@ksaprice_rabbitmq:8080//', celery_result_backend = db+postgresql://developer:user889@ksaprice_postgres:5432/airflow. These settings refer to either database or rabbitmq which are already running different containers - they do not refer to ip/url which runs celery and scheduler and I assuming it is this way because celery and scheduler runs on the airflow server.

My questions are:

  1. Reffering to point 1: Is there a simple way to run airflow webserver, airflow scheduler and airflow worker commands in the same Airflow container?
  2. Refferring to point 2: Is there a way in airflow.cfg to configure airflow scheduler and airflow worker to run in separate docker containers - and link them using docker network?

I am new bee to Airflow and Docker.

airflow docker celery
2 Answers

1- I did install all of these so its possible

2- optimized way is to install airflow(webserver) + backend-DB(Mysql) on one server, queuing(RabbitMQ) on another and Celery part on other set of servers. bellow I will mention something from the source and can be helpful to clarify things a bit better:

CeleryExecutor is one of the ways you can scale out the number of workers. For this to work, you need to setup a Celery backend (RabbitMQ, Redis, …) and change your airflow.cfg to point the executor parameter to CeleryExecutor and provide the related Celery settings.

Here are a few imperative requirements for your workers:

airflow needs to be installed, and the CLI needs to be in the path

Airflow configuration settings should be homogeneous across the cluster

Operators that are executed on the worker need to have their dependencies met in that context. For example, if you use the HiveOperator, the hive CLI needs to be installed on that box, or if you use the MySqlOperator, the required Python library needs to be available in the PYTHONPATH somehow

The worker needs to have access to its DAGS_FOLDER, and you need to synchronize the filesystems by your own means. A common setup would be to store your DAGS_FOLDER in a Git repository and sync it across machines using Chef, Puppet, Ansible, or whatever you use to configure machines in your environment. If all your boxes have a common mount point, having your pipelines files shared there should work as well

Source: https://airflow.readthedocs.io/en/1.10.6/howto/executor/use-celery.html

user_104996240
Related