Is there anyway to auto scale-out based on schedule not policy for dataproc in GCP (google cloud platform)

user_14791976 0
Viewed 97

It's straight forward. The system i am developing will be provisioned in dataproc (google cloud platform) And due to some business feature about our system, we can figure expected data processed in the future. So, i just want to do scale-out dataproc based on it.

Is there any idea to scale-out dataproc by scheduler or API implemented by our side ??

Thanks in advance.

google-cloud-dataproc google-cloud-platform
2 Answers

You can update the size of a Dataproc cluster using a gcloud dataproc clusters update or an equivalent API call in one of the Dataproc client libraries, see the documentation here. It also supports setting an auto scaling policy, where the cluster can grow and shrink based on the load on the cluster.

user_828040

You should use combination of Cloud scheduler and manual scale out method for this situation.

Create Cloud scheduler job runs manual scale out API call like below.

You must know about cron syntax to use cron scheduling for your scailing job.

enter image description here

In this scenario, You can only specify number of nodes your cluster uses, not number of incremental. This is a limitation of this solution. Below is example of API call.

PATCH /v1/projects/project-id/regions/us-central1/clusters/example-cluster?updateMask=config.worker_config.num_instances,config.secondary_worker_config.num_instances
{
  "config": {
  "workerConfig": {
  "numInstances": 4
},
"secondaryWorkerConfig": {
  "numInstances": 2
}
  },
  "labels": null
}

For a detailed description of Dataproc API, See here.

user_148350410
Related