We are trying to run Spark Submit job and we noticed that Spark Scratch Directories are created in /opt/mapr/spark/spark-1.6.1/bin when we use the following command to run Spark Submit jobs through YARN. We don't have access to create temp directories in /opt/mapr/spark/spark-1.6.1/bin. So we are trying to set the temp dir path with a folder we have write access using the conf property spark.local.dir.
./spark-submit --class org.apache.beam.runners.spark.examples.WordCount --master yarn --conf "spark.local.dir=/hadoop/landingzone/" /hadoop/landingzone/beam-runners-spark-bundled-spark.jar --runner=SparkRunner --inputFile=/hadoop/landingzone/pom.xml --output=counts
Exception Received:
IOException because we don't have write access to /opt/mapr/spark/spark-1.6.1/bin directory.
Warning Message:
In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN)
Could you please guide how to set scratch directory path when submitting jobs through YARN as spark.local.dir setting is not working.
Thanks