Does writing pyspak sql dataframe to disk and reading again speed up the further processing?

user_3884607 0
Viewed 92

Scenario1

  1. Pyspark sql dataframe df1 consisting of shape (7823061,26) has been made by doing lot of transformation in pyspark.
  2. Further transformation on this pyspark dataframe like group by and join takes a lot of time, like in hours.

Scenario2

  1. Save this dataframe on the disk using coaelesce and reading the dataframe again.
  2. Improves the performance of further transformations considerably.(like in seconds from hours)

Question Why does this improvement in performance happens?

pyspark apache-spark-sql
1 Answers

This is why Spark has Checkpointing feature.

Scenario 1 : If you look at the df1.explain(), it will have huge plan representing the DAG to create this dataframe. If at all the records of the df1 are available in memory or disk, then further operations will be fast. But if the dataset is huge, it is possible that some of the data will be evicted or lost. Due to which spark will now recompute each lost partition from the last successful DAG stage whose result is available (which can actually go back to starting point.) The method is also responsible for the fault tolerant feature.

Scenario2 : When write and read back , then check newDf.explain(), you will find that read itself is starting Point. Meaning now you have all the precomputed data as the starting point. Hence further actions just start from this stage and hence it is faster.

Again, scenario 1 is not very common. Generally I have faced it when there is a lot of data as well as lot of transformations over it.

Further more, read about checkpointing in spark.

user_70945200
Related