We are using Spark version 2.3.2 and I would like to use Spark's dynamic resource allocation on the YARN cluster. The prerequisite to use dynamic resource allocation is having the shuffle service enabled on all the executors/nodes in the cluster.
I do not have access to all the nodes in the cluster so I am not able to check the yarn-site.xml file for the shuffle related parameters in the nodes.
I am not sure if I am going in the right direction with my approach as listed below but please let me know the best way to find out if External Shuffle Service is enabled on my cluster or not.
I tried running the spark-shell with dynamic configuration enabled
spark.dynamicAllocation.enabledas well as enablingspark.shuffle.service.enabledbut when I print out all the SparkConf attributes and its values, it is not printing the keysspark.dynamicAllocation.enabledas well asspark.shuffle.service.enabled. This made me think that shuffle service may not be enabled and that's the reason I am not seeing those properties.I tried looking at the
yarn-site.xmlfile in the edge node to see if the propertyyarn.nodemanager.aux-servicesis set tospark_shuffleas well asyarn.nodemanager.aux-services.spark_shuffle.classtoorg.apache.spark.network.yarn.YarnShuffleServicebut I do not see those properties in the XML file. So I am thinking like Shuffle service might not have been enabled in our cluster.