spark-submit --files option not working as expecting
I am trying use following option for spark-submit
--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).
sh-4.2$ spark-shell --files etl_emr_test_config.json
..............................................
.............................................
..........................
..................................
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222)
Type in expressions to have them evaluated.
Type :help for more information.
scala> import org.apache.spark._
import org.apache.spark._
scala> SparkFiles.get("etl_emr_test_config.json")
res0: String = /mnt/tmp/spark-770e7981-2a38-4b12-950d-3519e70bdbe0/userFiles-afa53bd8-45c9-4c30-a923-feb2f0927117/etl_emr_test_config.json
scala> spark.read.text(SparkFiles.get("etl_emr_test_config.json")).show()
org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ip-100-69-166-111.ec2.internal:8020/mnt/tmp/spark-770e7981-2a38-4b12-950d-3519e70bdbe0/userFiles-afa53bd8-45c9-4c30-a923-feb2f0927117/etl_emr_test_config.json;
I was expecting etl_emr_test_config.json to be present in SparkFiles.get("etl_emr_test_config.json") path but it gives me error that file is not present