Spark catalog.clearCache and unpersisting RDD's

user_1671407 0
Viewed 832

I am new to spark and was looking for an option to clear the cache completely. On digging more i found that i have two options

SparkInstance.catalog().clearCache(); // removes all the tables from the in-memory cache

where as on the other hand i can also do

persistentRDDs.forEach((k,v) -> {
   v.unpersist(); // will unpersist all the RDD's
});

Now what i am trying to find and understand is which one would be a better option and why? I believe unpersisting RDD's will also remove all the data from memory that catalog() contains. Please correct me if i am wrong

spring-boot apache-spark java
1 Answers

if you are converting your dataframe to table/view by using the function createOrReplaceTempView then you can use cache table to cache that table. code as below :

df.createOrReplaceTempview("test") spark.catalog.cacheTable("test")

If you want to cache the dataframe itself than you can use df.cache() or df.persist(). For persist method you can pass parameter based on how you want to cache the dataframe.

If you want to uncache a dataframe you can use df.unpersist() but if you want to uncache the temp table that you created you can use spark.catalog.uncacheTable("test") and if you want to uncache all the temp tables that you have cached then you can use spark.catalog.clearcache().

you can also look at answer here to just get the reference : Temp table caching with spark-sql

user_48788480
Related