GCP Dataflow: using Zonal SSD with unknown zone

user_1783228 0
Viewed 120

Currently, we set region for our batch and stream Dataflow jobs, eg. europe-west1. When using HDD this is all nice.

But now I would like to test the pipelines with SSD, and according to worker-level-options, I can do it but I have to provide the Zone ex-ante.

Is it possible to use zonal SSD PD without specifying the zone, i.e. to use the zone of the workers (automatically)?

Note: I am aware that you can actually set the worker's zone, but I would like to avoid that and let them choose what is best within the region automatically.

apache-beam google-cloud-dataflow google-cloud-platform
1 Answers

I don't think Dataflow has options to [auto-]select the zone for SSD other than the worker_disk_type option.

Additionally, to improve the performance of your job (see this common error), you may switch to Dataflow Shuffle (should be turned on by default for all batch jobs) or streaming engine instead of using the appliance mode to minimize the local disk usage.

user_31758890
Related