Spark-submit --files option says that the files can be accessed using SparkFiles.get('files.txt')
So I wrote a simple program
from pyspark.sql import SparkSession
from pyspark import SparkFiles
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
print("testfile path : "+SparkFiles.get('testfile.txt'))
df=spark.read.text(SparkFiles.get('testfile.txt'))
df.show()
And then run using the below command :
spark-submit --master yarn --deploy-mode client --files testfile.txt testsubmit.py
From the logs I can see thaat the "testfile.txt" has been copied to hdfs://dummyIP.com:8020/user/root/.sparkStaging/application_1581404080152_0079/testfile.txt
20/03/05 04:41:08 INFO Client: Source and destination file systems are the same. Not copying hdfs://dummyIP.com:8020/hdp/apps/2.6.5.0-292/spark2/spark2-hdp-yarn-archive.tar.gz
20/03/05 04:41:08 INFO Client: Uploading resource file:/root/sumit/test/testfile.txt -> hdfs://dummyIP.com:8020/user/root/.sparkStaging/application_1581404080152_0079/testfile.txt
20/03/05 04:41:09 INFO Client: Uploading resource file:/usr/hdp/current/spark2-client/python/lib/pyspark.zip -> hdfs://dummyIP.com:8020/user/root/.sparkStaging/application_1581404080152_0079/pyspark.zip
But SparkFiles.get('testfile.txt') is trying to fetch 'testfile.txt' from hdfs://dummyIP.com:8020/tmp/spark-f7fedc0b-c3c7-4f6e-b72c-fc0618a03deb/userFiles-c90e2d49-c153-4945-bbe2-b006221002f9/testfile.txt
testfile path : /tmp/spark-f7fedc0b-c3c7-4f6e-b72c-fc0618a03deb/userFiles-c90e2d49-c153-4945-bbe2-b006221002f9/testfile.txt
Traceback (most recent call last):
File "/root/sumit/test/testsubmit.py", line 7, in <module>
df=spark.read.text(SparkFiles.get('testfile.txt'))
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 328, in text
File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://dummyIP.com:8020/tmp/spark-f7fedc0b-c3c7-4f6e-b72c-fc0618a03deb/userFiles-c90e2d49-c153-4945-bbe2-b006221002f9/testfile.txt;'
Also note that text() function will be executed on executor nodes as is mentioned the Spark Documentations. So it seems like SparkFiles.get('files.txt') is not reading from the same location where the --files is uploading it.