How to use RDD checkpointing to share datasets across Spark applications?

Viewed 387

I have a spark application, and checkpoint the rdd in the code, a simple code snippet is as follows(It is very simple, just for illustrating my question.):

@Test
  def testCheckpoint1(): Unit = {
    val data = List("Hello", "World", "Hello", "One", "Two")
    val rdd = sc.parallelize(data)
    //sc is initialized in the setup 
    sc.setCheckpointDir(Utils.getOutputDir())
    rdd.checkpoint()
    rdd.collect()
  }

When the rdd is checkpointed on the file system.I write another Spark application and would pick up the data checkpointed in the above code, and make it as an RDD as a starting point in this second application

The ReliableCheckpointRDD is exactly the RDD that does the work, but this RDD is private to Spark.

So,since ReliableCheckpointRDD is private, it looks spark doesn't recommend to use ReliableCheckpointRDD outside spark.

I would ask if there is a way to do it.

1 Answers
Related