Airflow Celery Worker celery-hostname

user_2471211 0
Viewed 801

On airflow 2.1.3, looking at the documentation for the CLI, airflow celery -h
it shows:

-H CELERY_HOSTNAME, --celery-hostname CELERY_HOSTNAME
                        Set the hostname of celery worker if you have multiple workers on a single machine

I am familiar with Celery and I know you can run multiple workers on the same machine. But with Airflow (and the celery executor) I don't understand how to do so.

if you do, on the same machine:

> airflow celery worker -H 'foo'
> airflow celery worker -H 'bar'

The second command will fail, complaining about the pid, so then:

> airflow celery worker -H 'bar' --pid some-other-pid-file

This will run another worker and it will sync with the first worker BUT you will get a port binding error since airflow binds the worker process to http://0.0.0.0:8793/ no matter what (unless I missed a parameter?).

It seems you are not suppose to run multiple workers per machines... Then my question is, what is the '-H' (--celery-hostname) option for? How would I use it?

airflow celery
1 Answers

The port that celery listens to is also configurable - it is used to serve logs to the webserver:

https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#worker-log-server-port

You can run multiple celery workers with different settings for that port or run them with --skip-serve-logs to not open the webserver in the first place (for example if you send logs to sentry/s3/gcs etc.).

BTW. This sounds strange by the way to run several celery workers because you can utilise the machine CPUS by using parallelism. This seems much more practical an easier to manage.

user_5167010
Related