I have the following code:
df = sql_context.sql("select * from table").cache()
first_df = df.where(df.id>10)
second_df = df.where(df.city = 'NY')
third_df = df.where(df.x == 5)
unioned_df = first_df.union(second_df).union(third_df)
unioned_df.format('csv').save(path)
Because my code has only one action (write to csv). Is there a point for caching df?
Please ignore the fact that this filters could be done all together.
I did it like this in purpose in order to understand how the cache mechanism work in the backgorund.