Is there a way to parallelize spark.read.load(string*) when reading many files?

user_11016321 0
Viewed 142

I noticed that in spark-shell (spark 2.4.4), when I do a simple spark.read.format(xyz).load("a","b","c",...), it looks like spark uses a single ipc client (or "thread") to load the files a, b, c, ... sequentially (they are path to hdfs).

Is this expected?

The reason I am asking is, for my case, I am trying to load 50K files, and the sequential load takes a long time.

Thanks

PS, I am trying to see it in the source code, but not sure if this is the one: https://github.com/apache/spark/blob/branch-2.4/sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala#L180

spark-shell apache-spark file-io performance
1 Answers

Might not be an exact "answer" to my original question, but I found out the reason for my particular case: from name node's audit log, it was found that there were some runaway jobs pegging name node, which greatly slowed down the rpc calls. After killing these bad jobs, the spark's load speed was greatly improved.

user_110163210
Related