Does Dataproc Auto Zone Placement randomly allocates zone

user_19345493 0
Viewed 20

I have used rest API to create my cluster in dataproc and have used auto zone placement to automatically choose a zone.

But somehow while running it, it will always choose the same zone and then sometimes cluster gets in ERROR state with "doesn't have enough resources to fulfill the request(resource type:compute)"

Manually creating in different zone allots me the resources, just the auto zone through REST API is not functioning well.

I have given zoneURI:'' in the request

My question is, is the auto zone placement random? And if not random is there a chance that all 3 zones in my case(europe-west2) don't have enough resources to fulfill the request?

google-cloud-dataproc google-cloud-platform
1 Answers

Dataproc Auto Zone is based on the actual resource availability, not random. It is possible that all 3 zones don't have enough resources to fulfill your request.

But note that it doesn't mean all zones are out of resources, it just means it couldn't fulfill your cluster's requirements for num of nodes * (machine type, disk types and sizes). Changing machine types and disk types can make a difference.

user_4042640
Related