Why spark submit --files option not working?

Viewed 459

spark-submit --files option not working as expecting

I am trying use following option for spark-submit

--files FILES Comma-separated list of files to be placed in the working directory of each executor. File paths of these files in executors can be accessed via SparkFiles.get(fileName).

sh-4.2$ spark-shell --files etl_emr_test_config.json
..............................................
.............................................
..........................
..................................

Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.4.0
      /_/

Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_222)
Type in expressions to have them evaluated.
Type :help for more information.

scala> import org.apache.spark._
import org.apache.spark._

scala> SparkFiles.get("etl_emr_test_config.json")
res0: String = /mnt/tmp/spark-770e7981-2a38-4b12-950d-3519e70bdbe0/userFiles-afa53bd8-45c9-4c30-a923-feb2f0927117/etl_emr_test_config.json

scala> spark.read.text(SparkFiles.get("etl_emr_test_config.json")).show()
org.apache.spark.sql.AnalysisException: Path does not exist: hdfs://ip-100-69-166-111.ec2.internal:8020/mnt/tmp/spark-770e7981-2a38-4b12-950d-3519e70bdbe0/userFiles-afa53bd8-45c9-4c30-a923-feb2f0927117/etl_emr_test_config.json;

I was expecting etl_emr_test_config.json to be present in SparkFiles.get("etl_emr_test_config.json") path but it gives me error that file is not present

0 Answers
Related