Im working on a streaming application and trying to unpersist a Dataframe,so which is better to use to clear cache unpersist() --async call or unpersist(true)--blocking call
which is better to use and why?, data size in the DF is nearly 150Gb. And what happens internally in both cases
df.unpersist() //df is a cached dataframe
val inputDf: DataFrame = readFile(spec, sparkSession) //read file from S3
or anyother source
val recreateddf = inputDf.persist()
Spark documentation
DataFrame unpersist()
DataFrame unpersist(boolean blocking)
https://spark.apache.org/docs/1.4.0/api/java/org/apache/spark/sql/DataFrame.html