How to increase the worker storage in cloud composer 2 which is max set to 10 GB

user_10124083 0
Viewed 97

We have recently migrated from Composer 1 to Composer 2. One of the task is heavily affected after this migration. The task runs using BigqueryOperator. The query processes 50TB of data.

Composer 1 Configuration :

Web server machine type
composer-n1-webserver-2 (2 vCPU, 1.6 GB memory)
  
Cloud SQL machine type
db-n1-standard-2 (2 vCPU, 7.5 GB memory)
  
Worker nodes
Node count
3   
Disk size (GB)
50
Machine type
e2-standard-4
Number of schedulers
1

The query use to take around 40 mins

Composer 2 Configuration:

Resources
Workloads configuration
Scheduler
4 vCPUs, 7.5 GB memory, 5 GB storage
Number of schedulers
2
Web server
2 vCPUs, 7.5 GB memory, 10 GB storage
Worker
4 vCPUs, 16 GB memory, 10 GB storage
Number of workers
Autoscaling between 4 and 8 workers

The same query takes around 1 hour 40 mins.

Does worker storage(Disk) reduction from 50GB(Composer 1)to 10 GB(Composer 2) is affecting the query run.

Does worker nodes play any role query computation or they just takes the tasks from queue and submit the query to Bigquery(in this case)?

airflow-2.x google-cloud-composer google-cloud-platform
1 Answers

This is a limitation as Composer 2 uses GKE Autopilot. Refer to this document for details of Composer 2.

Composer workers are intended to be used for launching and/or polling for the status of work launched elsewhere.

You can use ephemeral-storage limits on the KubernetesPodOperator e.g. k8s.V1ResourceRequirements(limits={"ephemeral-storage": "2Gi"})

Ephemeral volumes are designed for these use cases. Because volumes follow the Pod's lifetime and get created and deleted along with the Pod, Pods can be stopped and restarted without being limited to where some persistent volume is available.

user_169293580
Related