Tried running the below code snippet in spark 3.0.2
val df = spark.read.parquet(file_path)
val df1 = df.select(col1, col2).filter(col2 > 10).cache()
df1.count
However when i tried the below code, could see its not leveraging the cached data.
df.select(col2).filter(col2 > 10).count .
My understanding is that ResolveMissingReferences analyser rule will add the missing attribute and leverage the cached data.
Any advice is highly appreciated ?