Check if External Shuffle Service is enabled on YARN Cluster

Viewed 464

We are using Spark version 2.3.2 and I would like to use Spark's dynamic resource allocation on the YARN cluster. The prerequisite to use dynamic resource allocation is having the shuffle service enabled on all the executors/nodes in the cluster.

I do not have access to all the nodes in the cluster so I am not able to check the yarn-site.xml file for the shuffle related parameters in the nodes.

I am not sure if I am going in the right direction with my approach as listed below but please let me know the best way to find out if External Shuffle Service is enabled on my cluster or not.

  1. I tried running the spark-shell with dynamic configuration enabled spark.dynamicAllocation.enabled as well as enabling spark.shuffle.service.enabled but when I print out all the SparkConf attributes and its values, it is not printing the keys spark.dynamicAllocation.enabled as well as spark.shuffle.service.enabled. This made me think that shuffle service may not be enabled and that's the reason I am not seeing those properties.

  2. I tried looking at the yarn-site.xml file in the edge node to see if the property yarn.nodemanager.aux-services is set to spark_shuffle as well as yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService but I do not see those properties in the XML file. So I am thinking like Shuffle service might not have been enabled in our cluster.

0 Answers
Related