How to increase cloud composer 2 worker storage limit beyond 10GB when using KubernetesPodOperator?

user_905741 0
Viewed 420

I have configured cloud composer 2 using terraform with the following configuration:

workloads_config {
      worker {
        cpu        = 2
        memory_gb  = 6
        storage_gb = 10
        min_count  = 1
        max_count  = 4
      }
    }

I cannot set storage_gb higher than 10, either in terraform or in the GCP console. In terraform, I get an error saying I can't set the value above 10GB. In the console, it's not possible to enter a value greater than 10.

I have one job using the KubernetesPodOperator which is failing because it hits the disk limit.

I can see the pod was evicted with the following error:

Pod ephemeral local storage usage exceeds the total limit of containers 10239Mi.

The job worked OK on Composer 1 and I don't recall having to configure the storage limit there but Composer 1 has a lot of differences in how the environment is setup.

I have not been able to find docs on how to increase this limit in Composer 2.

Things I will try next:

  • try asking for more storage using pod resource requests
  • see is there a way to configure the limits on the cluster outside of composer

I have not been able to find docs on how to change the 10GB limit besides this doc which shows how to change the storage for a worker but doesn't mention the 10GB limit.

Is there a standard way to increase the worker limit above 10GB?

airflow-2.x google-cloud-composer airflow
1 Answers

This is a limitation as Composer 2 uses GKE Autopilot. You may refer to this Composer 2 for details of Composer 2.

Composer workers are intended to be used for launching and/or polling for the status of work launched elsewhere. It is not meant to be used for local processing or for scratch computation.

If you intend to have a scratch space to work on files or do local processing, you may launch other Kubernetes pods to do so and then use KubernetesPodOperator or you may also try GKEPodOperator since the KubernetesPodOperator didn't work for you.

user_177203540
Related