I'm working with spark SQL dataframes and am having an issue with persisting to speed up later computations. Specifically, when calling persist(StorageLevel.MEMORY_AND_DISK) and then subsequently checking in the Spark UI's 'Storage' tab I'm able to see the RDDs being cached but the storage level always shows Memory Deserialized 1x Replicated and the "Size on Disk" column shows 0.0 B for all RDDs.
I've also tried MEMORY_AND_DISK_SER but get the same results. I'm curious if anyone has seen this or if I'm doing something incorrectly here. Reviewing the spark docs, shows that calling either cache() or persist() on a dataframe defaults to a storage level of MEMORY_AND_DISK and using the cacheTable method in SQLContext states that it Caches the specified table in-memory.
For some additional information, a general skeleton of my program's flow is:
// Here computeHeavyMethod is some code that returns a DataFrame
val tableData = computeHeavyMethod().persist(StorageLevel.MEMORY_AND_DISK)
tableData.write.mode(SaveMode.Overwrite).json(outputLocation)
tableData.createOrReplaceTempView(tableName)
spark.sql("Some sql statement that uses the table created above")