The difference on reading files in PySpark between reading the whole directory then filtering and reading a part of the directory?

user_13703437 0
Viewed 349

Suppose I have a data model that runs daily and the sample HDFS path is

data_model/sales_summary/grass_date=2021-04-01

If I want to read all the models in Feb and March, what is the difference if I read in the following two ways:

A:

spark.read.parquet('data_model/sales_summary/grass_date=2021-0{2,3}*')

B:

spark.read.parquet('data_model/sales_summary/').filter(col('grass_date').between('2021-02-01', '2021-03-30'))

Are these two reading methods equivalent? If not, under what circumstances which one can be more efficient?

pyspark apache-spark-sql apache-spark
2 Answers

Spark will do a partition filter when reading the files, so the performance of the two methods should be similar. The query plans below show how the partition filters are used in the filescan operation.

spark.read.parquet('data_model/sales_summary/grass_date=2021-0{2,3}*').explain()
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet [id#18] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/tmp/data_model/sales_summary/grass_date=2021-02-21, file:/tmp/data_model/..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<id:int>

spark.read.parquet('data_model/sales_summary/').filter(F.col('grass_date').between('2021-02-01', '2021-03-30')).explain()
== Physical Plan ==
*(1) ColumnarToRow
+- FileScan parquet [id#24,grass_date#25] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex[file:/tmp/data_model/sales_summary], PartitionFilters: [isnotnull(grass_date#25), (grass_date#25 >= 18659), (grass_date#25 <= 18716)], PushedFilters: [], ReadSchema: struct<id:int>

But note that the partitioning column will be missing from the dataframe if you use the first method to read the files, so you'd probably prefer the second method.

user_141657300

We should use First one (A) is right .

A- We are selecting specific folders (we are reading required data only).

B - We are reading all data and then applying filter ( here we are reading all data which is costly ).

user_60858030
Related