Is the checkpoint requires for the delta lake merge operation in a streaming job

user_10017348 0
Viewed 90

I have an understanding that for a spark streaming merge it's helpful to have a checkpoint location specified to not process stuff twice on the job restart (even if the operation is idempotent and ins't mentioned in example notebook). Is it correct?

delta-lake databricks apache-spark upsert
1 Answers

If you don't specify the location of the checkpoint, each time all the data will be reprocessed.

user_111714010
Related