Is it efficient to read from a LIST of FILES instead of a PATH in Spark?

user_9885786 0
Viewed 779

I am using pyspark in azure databricks. And need to load thousands of files as a list of files. "Multi depth partitioning" is used, which makes difficult to use the base path to read the files.

Indeed, this multi-depth partitions results in nested directories which trigger this error :

AnalysisException: Unable to infer schema for CSV. It must be specified manually.;

Hence, we are reading everything as a list of files and I would like to know if the performance is the same when you read files using :

1.

spark.read.format('csv').load('/mnt/article/2021/08/09')

vs

2.

spark.read.format('csv').load([
        '/mnt/article/2021/08/09/test.csv',
        '/mnt/article/2021/08/09/test2.csv',
        '/mnt/article/2021/08/09/test3.csv'
    ])

vs

3.

spark.read.format('csv').load(['/mnt/article/*/*/*/])

For some reasons, we don't want to use the third one : spark.read.format('csv').load(['/mnt/article/*/*/*/) but we might reconsider if the second one is really not efficient.

Many thanks in advance for any observations or suggestions!

pyspark azure-databricks apache-spark partitioning
1 Answers

You should try it by yourself, it is a good exercise.

However, I would say 2nd option is slightly faster because it does not require extra ls.

But I am not even sure of that because Spark checks if the files are leaves or not. It probably depends on the implementation of the connector. cf: def allFiles(): Seq[FileStatus]

Except if you have thousands of files on a file system where ls costs (typically a cloud provider where an ls is an HTTP request). It should not make a difference and you should choose the clearest option from a business point of view. Which is the 1st option you offered.

user_72761070
Related