Can we create a Hadoop Cluster on Dataproc with 0%-2% of HDFS?

user_3256598 0
Viewed 80

Is it possible to create a Hadoop cluster on Dataproc with no or very minimal HDFS space by setting dfs.datanode.du.reserved to about 95% or 100% of the total node size? The plan is to use GCS for all persistent storage while the local file system will primarily be used for Spark's shuffle data. Some of the Hive queries may still need the scratch on HDFS which explains the need for minimal HDFS. I did create a cluster with a 10-90 split and did not notice any issues with my test jobs.

  1. Could there be stability issues with Dataproc if this approach is taken?
  2. Also, are there concerns with deleting the Data Node daemon from Dataproc's worker nodes, thereby using the Primary workers as compute only nodes. The rationale is that Dataproc currently doesn't allow a mix of preemptible and non preemptible secondary workers. So want to check if we can repurpose primary workers as compute only non-PVM nodes while the other secondary workers can be compute only PVM nodes.
hdfs google-cloud-dataproc apache-spark hadoop
1 Answers

I am starting a GCP project and am well-versed enough in AZURE and AWS less so, but know enough there having done a DDD setup.

What you describe is similar to AWS setup and I looked recently here: https://jayendrapatil.com/google-cloud-dataproc/

My impression is you can run without HDFS here as well - 0%. The key point is that performance with a suite of jobs will - like also for AWS & AZURE - benefit from writing to and reading from ephemeral HDFS, as it is faster than Google Cloud Storage. I cannot see stability issues; I can use Spark now without HDFS if I really want.

On the 2nd question, stick to what they have engineered. Why try and force things? On AWS we lived with the limitations on scaling down with Spark.

user_69339930
Related