How to load local file on the app driver with spark cluster?

user_5286990 0
Viewed 271

I have run a spark cluster(standalone cluster mode), and a scala program. The scala program may accept some csv file, and load csv into dataset. My code is like below:

    val sparkSession = SparkSession.builder
      .appName("app")
      .master("spark://172.17.0.2:7077")
      .config("spark.scheduler.mode", "FIFO")
      .config("spark.memory.fraction", 0.3)
      .config("spark.sql.shuffle.partitions", 1)
      .config("spark.sql.debug.maxToStringFields", 600)
      .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
      .getOrCreate
    sparkSession.sparkContext.addFile("file:///home/liusf/i1.csv")
    val ds = sparkSession.read.csv(SparkFiles.get("i1.csv"))
    ds.show(1)

but it throws a FileNotFoundException,error message is below

18:17:32  WARN [task-result-getter-0] [o.a.spark.scheduler.TaskSetManager:69] 
Lost task 0.0 in stage 0.0 (TID 0) (172.17.0.3 executor 1): 
java.io.FileNotFoundException: File file:/tmp/spark-647c0146-5ba5-433f-9555-6b64a16231a7/userFiles-61074e0d-9edf-4c75-9857-cb9db63b3925/i1.csv does not exist

while in local model(master set as local[1]), it work as expected.
I've searched on google and found two sugession, one sugession is "copy local files to all work nodes", and the other is " use nfs or hdfs to share files".
Is there a simple way that make the program work without introducing extra technology?

apache-spark scala
1 Answers

By nature of clusters the job can be executed on any of the worker nodes. Unless you copy the file or store the file on a location that can be accessed by all workers spark won't be able to locate the file.

The simple way to make the program work is to scpthe file to the desired location on all worker nodes.

user_41862460
Related