pyspark optimization for exceptAll

user_17613515 0
Viewed 102

I have a simple PySpark code using default Spark standalone config.

There are 2 files both around 2GB in size:

  • df1 - load file1
  • df2 - load file2

then find unique data from df1 dataframes:

df3 = df1.exceptAll(df2)
df3.show()

Is there a way to make this code run faster?

pyspark python
1 Answers

Which columns are you using to find the uniqueness? You may not need to check all the columns. You can select the columns used for uniqueness and then use exceptAll to increase performance.

user_46595300
Related