If I checkpoint a DataFrame like the below. How can I read it back in?
df1 = spark.createDataFrame([('Abraham','Lincoln')], ['first_name', 'last_name'])
df1.checkpoint()
Something like....
reload = spark.read.something('checkpoints/87b411a8-19e3-402a-86a7-cfac0a4a6d14/rdd-40/*')
I see that the checkpoint file that is written to the checkpoints folder is partitioned but can't tell what the file type is.