How to read files uploaded by spark-submit on Kubernetes

user_2930152 0
Viewed 2570

I have Spark Jobs running on Yarn. These days I'm moving to Spark on Kubernetes.

On Kubernetes I'm having an issue: files uploaded via --files can't be read by Spark Driver.

On Yarn, as described in many answers I can read those files using Source.fromFile(filename).

But I can't read files in Spark on Kubernetes.

  • Spark version: 3.0.1
  • Scala version: 2.12.6
  • deploy-mode: cluster
  • submit commands 
    $ spark-submit --class <className> \
      --name=<jobName> \
      --master=k8s://https://api-hostname:6443 \
      ...
      --deploy-mode=cluster \
      --files app.conf \
      --conf spark.kubernetes.file.upload.path=hdfs://<nameservice>/path/to/sparkUploads/ \
      app.jar

After executing above command, app.conf is uploaded to hdfs://<nameservice>/path/to/sparkUploads/spark-upload-xxxxxxx/,

And in Driver's pod, I found app.conf in /tmp/spark-******/ directory, app.jar as well.

But Driver can't read app.conf, Source.fromFile(filename) returns null, there was no permission problems.

Update 1

In Spark Web UI->"Environment" Tab, spark://<pod-name>-svc.ni.svc:7078/files/app.conf in "Classpath Entries" menu. Does this mean app.conf is available in classpath?

On the other hand, in Spark on Yarn user.dir property was included in System classpath.

I found SPARK-31726: Make spark.files available in driver with cluster deploy mode on kubernetes

Update 2

I found that driver pod's /opt/spark/work-dir/ dir was included in classpath.

but /opt/spark/work-dir/ is empty on driver pod whereas on executor pod it contains app.conf and app.jar.

I think that is the problem and SPARK-31726 describes this.

Update 3

After reading Jacek's answer, I tested org.apache.spark.SparkFiles.getRootDirectory().

It returns /var/data/spark-357eb33e-1c17-4ad4-b1e8-6f878b1d8253/spark-e07d7e84-0fa7-410e-b0da-7219c412afa3/userFiles-59084588-f7f6-4ba2-a3a3-9997a780af24

Update 4 - work around

  • First, I make ConfigMaps to save files that I want to read driver/executors
  • Next, The ConfigMaps are mounted on driver/executors. To mount ConfigMap, use Pod Template or Spark Operator
apache-spark kubernetes scala
2 Answers

--files files should be accessed using SparkFiles.get utility:

get(filename: String): String

Get the absolute path of a file added through SparkContext.addFile().

user_13053440

I found the another temporary solution in spark 3.3.0

We can use flag --archives. The files without tar, tar.gz, zip are ignored unpacking step and after that they are placed on working dir of driver and executor.

Although the docs of --archive don't mention executor, I tested and it's working.

user_100736700
Related