Scheduling start/stop DataProc clusters

user_13278906 0
Viewed 464

From the provided docs for dataprocs it says:

To help avoid incurring Google Cloud charges for an inactive cluster, use Dataproc's Cluster Scheduled Deletion feature when you create a cluster. This feature provides options to delete a cluster:

  • after a specified cluster idle period
  • at a specified future time
  • after a specified period that starts from the time of submission of the cluster creation request

However what i'm looking for is something that allows me to schedule the cluster activity automatically for example i want the cluster to be active only during working days and working hours.
This way i can avoid incurring in additional costs without having to manually stop (or rather only delete as it seems from DataProc documentation) and recreate the cluster every day.

For this purpose in Compute Engine i'm able to create a scheduling for the VMs of the cluster to stop them.
although, by using this method, the VMs are stopped through this schedule i still see the cluster "running" in DataProc.

Am i still incurring in costs or not? if yes are there any similar alternatives to schedule in DataProc?

As a bonus question is it possible to apply a schedule to a cluster with an autoscaling policy?

google-compute-engine google-cloud-dataproc google-cloud-platform
1 Answers

The best way to meet your requirements is to use Cloud Composer with Dataproc start and stop APIs.

If you stop the underlying VMs, Dataproc will also stop charging for the stoped VMs, because Dataproc billing is based on core hour. But it is not the safest and robust way to save cost, I believe Dataproc stop has some logic to make sure the stop is safe. We should definitely prefer Dataproc stop API over stopping VMs directly. But note that either way you will still pay for persistent disk for stopped VMs. So consider keeping your input and output data in GCS, create the cluster when you need to do data processing, delete after that. Recreating cluster will also get you the latest updates and bug fixes which restarting cluster doesn't offer.

Autoscaling won't remove master nodes, and as a best practice, your policy should avoid scaling primary workers and only scale secondary workers, because primary workers need to keep HDFS data (and shuffle data if EFM is enabled). So if you want to stop the cluster completely to avoid cost, autoscaling is not a good choice.

user_4042640
Related