I'm running a spark job that takes an input, which is generated by the same previous job. Right now the job outputs the results to HDFS for the next run to read in, is there a way to cache the output of each job in spark so that the following run won't have to read from HDFS?
Update: or is it possible for spark to share RDD among different applications?