VACUUM/OPTIMIZE Effect on Autoloader Checkpoints

user_11137562 0
Viewed 321

I'm using Databricks Autoloader to incrementally stream from a Delta Lake table into a SQL database. If an OPTIMIZE or VACUUM statement is ran against the Delta table, new files are added/subtracted.

My question is, will the autoloader checkpoint discount these optimized files on the next stream? Or will my entire Delta table be streamed into SQL because autoloader doesn't recognize it's already processed the data?

azure-databricks databricks apache-spark
1 Answers

As long as you specify the format of the readStream correctly, the autoloader checkpoint will disregard all aggregated files created by OPTIMIZE command. In this case, the code should be started as follows. df.readStream.format('delta')

user_112893860
Related