Problem is the same the title. We sometime wait about 1hour. This thing make our develop experience become too bad.
Composer version is composer-2.0.4-airflow-2.2.3 .
We have 17 DAGs.
Scheduler parse DAGs fast. So, We expect that workers of composer doesn’t sync DAGs with GCS FUSE.
Are there other reason？ What should we do to solve this problem？
Our GKE Workload Configuration is follow the picture.
According to the configuration, I would suggest you increase the resources. Generally in Cloud Composer 2, the GKE workloads like Scheduler and Workers have their resources limited to the resources defined. Sometimes lack of CPU and memory resources also lead to delay in synchronization. You can monitor your DAG’s to increase and decrease the resources according to the requirement as mentioned in this documentation.
There are many possible causes for delayed synchronization. You can follow this documentation for handling larger numbers of DAG’s. For more information on tuning Cloud Composer performance, you can check this link.
