We have recently shifted to airflow 2.0. I wanted to understand the procedure to set up multiple schedulers. I tried to see various sources including the procedure which uses astronomer cli. But we are not using that. For other sources including airflow documentation, while they have mentioned that it is possible to do so and given some details about the database requirements, they have not given any details on how exactly one goes about setting multiple schedulers (like steps or commands).
Do we just schedule a new scheduler using
airflow scheduler ?
or do we have to configure something in airflow.cfg or the docker file?
Also how do we verrify that a new scheduler has been set up? Using astronomer cli, it is possible to easily kill the scheduler. But I was not able to find anything when we are not using it.
How to set up multiple schedulers for airflow
Viewed 2405
1 Answers
Yes, if you are using Helm Chart, you can use replicas for scheduler to 2.
If you are using just VM: both the VMs can run multiple scheduler process just by running airflow scheduler. No additional configurations are required.
You can verify in that Airflow Webserver by going to Browse -> Jobs page and filter by JobType as SchedulerJob that are in state running.
Related