I have a few DAGs that only run for a few hours a day, after which the Cloud Composer cluster does nothing. I'm trying to understand the worker pool auto-scaling mechanism:
concurrency=10, will Composer run on those 3 nodes until the DAG hints at it that it needs for resources, at which point it auto scales to 10 nodes? Then will it destroy those additional 7 nodes when the DAG is done?Cloud Composer doesn't natively support auto-scaling, but you can hack it by changing airflow config and change the underlying k8s settings.
There's a great article on how to do it here: https://medium.com/traveloka-engineering/enabling-autoscaling-in-google-cloud-composer-ac84d3ddd60
I have decided to elaborate little bit more to @SNeumann answer. As for now, Composer doesn't support Kubernetes clusters with autoscaling enabled. This is due to there's a complex conversation around Kuberenetes resource requests and how those can't be estimated for Airflow pods at this time.
Feature requests have already been filed: 1, 2 to support worker autoscaling based on DAG demand. I suggest subscribing it to keep updated as you'll receive e-mails regarding its progress.
According to the official documentation:
the Airflow scheduler and workers are in the same GKE cluster, the schedulers and workers won't work properly if the competition results in resource starvation.
For now, you need to enable autoscaling on node level and write .yaml, then update minReplicas: 1 and maxReplicas: 50 like in the Medium tutorial posted by @SNeumann.
I hope you find the above pieces of information useful.
Environment scaling is now natively supported in Cloud Composer 2:
Cloud Composer 2 brings environments that scale automatically based on the demands of your workflows. For more information about Cloud Composer 2, see Major versions of Cloud Composer, Environment scaling, and Pricing pages in the documentation.
Example using gcloud command:
gcloud beta composer composer environments update <ENVIRONMENT_NAME> \
--location <LOCATION> \
--min-workers <WORKERS_MIN> \
--max-workers <WORKERS_MAX>