I have a large dataframe that has been cached like
val largeDf = someLargeDataframe.cache
Now I need to union it with a tiny one and cached it again
val tinyDf = someTinyDataframe.cache
val newDataframe = largeDf.union(tinyDf).cached
tinyDf.unpersist()
largeDf.unpersist()
It is very inefficient since it need to re-cached all the data again. Is there any efficient way to add a little amount of data to a large cached dataframe?
After reading Teodors's explanation, I know that I can't unpersist the old dataframe before I do some action on my new dataframe. But what if I need to do something like this?
def myProcess(df1: Dataframe, df2: Dataframe): Dataframe{
val df1_trans = df1.map(....).cache
val df2_trans = df2.map(....).cache
doSomeAction(df1_trans, df2_trans)
val finalDf = df1_trans.union(df2_trans).map(....).cache
// df1_trans.unpersist()
// df2_trans.unpersist()
finalDf
}
I want my df1_trans & df2_trans to be cached to improve the performance inside the function since they will be called more than once, but the dataframe I need to return in the end is also constructed by df1_trans & df2_trans, if I can't unpersist them before leaving the function, I can never find other place to do this, however, if I unpersist them, my finalDf will not benefit from cache.
What can I do in this situation? Thanks!