My Spark/Scala job is processing data in a loop. I need to use the result of the previous loop to process information. I was caching the results, but the issue is that the spark plan for the cached DataFrame grows too large and my job errors out. While I have found a back up solution of writing to and reading from S3, it is time consuming. So, if there is a way to utilize the cache without growing the spark plan I would like to try it out. Or if there is another way to keep track of the computed data within the Spark program that would also help.
2 Answers
You can use Dataset.checkpoint() method at the end of each iteration.
It will persist your current RDD state and truncate the plan.
You need to manually set a checkpoint directory before hand using SparkContext.setCheckpointDir(), since it's unset by default.
You just need to increase the spark.driver.memory to a value that is more than the size of the plan
Related