Im using spark-2.1.1-bin-hadoop-2.7 standalone mode (cluster of 4 workers, 120g memory, 32 cores total)
Although I defined spark.local.dir conf param to write to /opt, spark worker keep writing to /tmp dir, for example /tmp/spark-e071ae1b-1970-47b2-bfec-19ca66693768
Is there a way to tell spark worker not write to /tmp dir?
As per the spark documentation, few environment variables will override the property 'spark.local.dir', please try checking these environment variables
Quoting from the documentation:
spark.local.dir
Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. Note: This will be overridden by SPARK_LOCAL_DIRS (Standalone), MESOS_SANDBOX (Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager.
