Why are persistent Dataproc clusters not recommended?

user_331393 0
Viewed 406

I am considering running a persistent GCP Dataproc cluster that hosts a Hive server, which will provide a HiveQL interface for querying and updating long-lived data stored in Google Cloud Storage, accessed via the Cloud Storage connector.

I am reading the following documentation: https://cloud.google.com/architecture/hadoop/hadoop-gcp-migration-overview#moving_to_an_ephemeral_model

Advantages of ephemeral clusters are listed, but also the following warning:

If you can't accomplish your work without a persistent cluster, you can create one. This option may be costly and isn't recommended if there is a way to get your job done on ephemeral clusters.

Aside from not enjoying the listed advantages of ephemeral Dataproc clusters, are there any additional downsides/gotchas to running persistent Dataproc clusters?

My primary motivation for maintaining a persistent cluster is to avoid any admin overhead of having to re-create clusters. The cluster needs to be available to serve Hive clients indefinitely; there's no natural end-of-cluster date.

Edit: To be clear, I'm worried that a long-running persistent cluster might accumulate faults over time, analogous to memory leaks.

google-cloud-dataproc
3 Answers

When you have a persistent cluster two things happen:

  • First, you will try to run as many processes you can on it just to optimize usage.

    If you're on a physical hadoop/spark cluster this is a good idea since hardware is costly, but you will end parsing logs to find out which department or use case is actually using all your cluster capacity.

  • Second, your cluster will be idle some time and you'll be paying for machines that do nothing until you need to run a job.

    Since you're in the cloud you can create a dataproc cluster just for the job you need to run and scrap it when the job finishes (just storing the results in cloud storage).

IF you run that cluster (and the job) inside its own project, you will be able to identify easily costs for each center/department, etc. without having to parse log files.

And, of course, you can save money just shutting down the machines when there is no calculation to do.

Shutting down stuff when is not needed is what makes cloud cost-effective.

If you just need to make ad-hoc (throwaway) queries, you can make BigQuery retrieve data directly from cloud storage. Remember BQ will not cache queries against storage -> do not use that queries to feed a dashboard application.

Usually using bigquery for data access / reporting is cheaper (and faster) than having a dataproc cluster running 24x7... unless you have a lot of jobs all day and there is very little 'free' time between them... or you cannot modify the application that queries Hive.

user_67603720

Persistent / long-running Dataproc clusters are definitely supported, but they have their own downsides (cost, software upgrade, etc). Check this doc for best practices of using long-running Dataproc clusters.

user_4042640

In addition to not optimal resource usage, the main disadvantage of the persistent cluster is that you can not change its configuration and update software running on it.

You should have an automation that re-creates cluster periodically to pick up new software fixes and updates (i.e. Dataproc image versions), and apply new configuration. This will allow you to reduce administrative cost, not increase it, because administering a long running snowflake cluster is much harder than ephemeral.

user_32276930
Related