Concurrent update to delta lake table via multiple jobs

Question

I have a delta table, where multiple jobs via databricks can merge/upsert data into the delta table concurrently.

How can I prevent from getting ConcurrentAppendException?

I cannot use this solution, as the incoming changes can be a part of any partition and I cannot filter any partition.

Is there a way to check whether the Delta table is being appended/merged/updated/deleted and wait until its completed and then we acquire the locks and start the merge for the second job?

Just FYI, these are 2 independent Azure Datafactory jobs trying to update one delta table.

Cheers!

user_277035 · Answer

You should handle concurrent appends to Delta as any other data store with Optimistic Offline Locking - by adding application-specific retry logic to your code whenever that particular exception happens.

Here's a good video on inner workings of Delta.

Concurrent update to delta lake table via multiple jobs

1 Answers