Why Apache Spark ML on Dataproc Trains Only on Single Node?

Question

I'm new to distributed ML and currently doing my personal project

I train my model using PySpark on Cloud Dataproc and build the pipeline like the code below

spark = SparkSession.builder.appName('sparkify-train').getOrCreate()
df = spark.read.parquet(path)

gbt = GBTClassifier()

paramGrid = ParamGridBuilder() \
    .addGrid(gbt.maxDepth, [4,8,12]) \
    .addGrid(gbt.maxIter, [5,10,15]) \
    .addGrid(gbt.featuresCol, ["features_1", "features_2"]) \
    .build()

tvs = TrainValidationSplit(estimator=gbt,
                           estimatorParamMaps=paramGrid,
                           evaluator=BinaryClassificationEvaluator(),
                           trainRatio=0.75)
tvs.fit(df)

Note: I don't show feature extraction and other stuff on the code above

But on CPU utilization chart, the master node is the only busy node instead of three workers node

I create Dataproc cluster using cloud shell with command below. This command is generated from Dataproc create cluster UI.

gcloud beta dataproc clusters create sparkify --enable-component-gateway --region asia-southeast2 --subnet default --zone asia-southeast2-c 
--master-machine-type custom-1-3840 --master-boot-disk-type pd-ssd --master-boot-disk-size 50 --num-workers 3 --worker-machine-type custom-1-3840 --worker-boot-disk-type pd-ss
d --worker-boot-disk-size 30 --image-version 2.0-ubuntu18 --optional-components JUPYTER,ZEPPELIN --project wskt-trek

and this is the cluster properties from command above:

I just wondering is there anything wrong in my code? Or are there any settings I miss? I thought all of the workers CPU should have high utilization. Thanks!

user_5895154 · Answer

The cluster is well configured , in your case you are using a notebbok , a client mode is established in this case, I advice you either:

You create a spark cluster with more than one master (2 masters and 3 workers for examle ) which is not a good practice (you will understant in the 3 rd point why ).
You can use spark-tensorflow-distributor which is an open-source native package in TensorFlow that helps users do distributed training with TensorFlow on their Spark clusters. It is built on top of tensorflow.distribute.Strategy, which is one of the major features in TensorFlow 2. For detailed API documentation, see docstrings. For general documentation about distributed TensorFlow, see Distributed training with TensorFlow.
The most import point is this: don't limit your vision in this level , I mean at the level of working with notebook and see the behavior of spark cluster, why ? Well after trainning your model you need to save it as a pmml file then deploy it , spark will work fine. You can refer to this simple example and try to deploy your model and compare spark cluster behavior in Dataproc to the previous one when using notebbok client mode.

Why Apache Spark ML on Dataproc Trains Only on Single Node?

1 Answers