Efficient way to join a cached spark dataframe with other and cache again

Viewed 6445

I have a large dataframe that has been cached like

val largeDf = someLargeDataframe.cache

Now I need to union it with a tiny one and cached it again

val tinyDf = someTinyDataframe.cache
val newDataframe = largeDf.union(tinyDf).cached
tinyDf.unpersist()
largeDf.unpersist()

It is very inefficient since it need to re-cached all the data again. Is there any efficient way to add a little amount of data to a large cached dataframe?


After reading Teodors's explanation, I know that I can't unpersist the old dataframe before I do some action on my new dataframe. But what if I need to do something like this?

def myProcess(df1: Dataframe, df2: Dataframe): Dataframe{
    val df1_trans = df1.map(....).cache
    val df2_trans = df2.map(....).cache

    doSomeAction(df1_trans, df2_trans)

    val finalDf = df1_trans.union(df2_trans).map(....).cache
    // df1_trans.unpersist()
    // df2_trans.unpersist()
    finalDf
}

I want my df1_trans & df2_trans to be cached to improve the performance inside the function since they will be called more than once, but the dataframe I need to return in the end is also constructed by df1_trans & df2_trans, if I can't unpersist them before leaving the function, I can never find other place to do this, however, if I unpersist them, my finalDf will not benefit from cache.

What can I do in this situation? Thanks!

3 Answers
Related