Does partition filter order has significant performance impact on spark read

user_5246271 0
Viewed 63

Example : table XXXX is partition by column1 and column2

           reading table XXXX by where clause Column1 and column2
           option 1: select xxxx where column1 ="" and column2 =""
           option 2: select xxxx where column2 ="" and column1 =""
          
   does option 1 and option 2 has different read performance because of filter order and table partitionBy column1 and column2?

i have checked DAG for both case which is having partitionFilter in filescan.

question is 
option1 : where(column1 ="" and column2 ="")
option 2: where(column2 ="" and column1 ="" ) 
does it have differenet read performance beacuse of table patition by column1, column2?
apache-spark
1 Answers

I would say it should make not a difference as ultimately the mapping (ie the total amount of processing time added to fetch the first partition and then the second) will happen the same either way. Basically ...

t(column1) + t(column2) = t(column2) + t(column1)

but this is a tricky question that should be tested with different cardinalities and formats for each column.

user_25211760
Related