Deadlock with concurrent Celery workers in Django

user_1467730 0
Viewed 996

I have dozens of concurrent Celery workers creating, updating, and deleting data from various models in my Django application. I'm running into the following error:

deadlock detected
DETAIL:  Process 3285070 waits for ShareLock on transaction 559341801; blocked by process 3285058.
Process 3285058 waits for ShareLock on transaction 559341803; blocked by process 3285070.
HINT:  See server log for query details.

And these are the blocks of code where it's throwing the error:

with transaction.atomic():
    for score in list_of_scores_to_update:
        score.save()

time_delta = timezone.now() - timezone.timedelta(minutes=2)
with transaction.atomic():
    Score.objects.select_for_update(of=('self'), skip_locked=True).filter(checked_date__lte=time_delta).delete()

How can I prevent these deadlocks by acquiring a write lock on all of the rows prior to updating them?

django-rest-framework deadlock postgresql django-models django
2 Answers

The best way to prevent deadlocks is to

  • keep your transactions short

  • keep your transactions small (change as few rows as possible)

Now it may be that your requirements make this difficult. Then you can avoid deadlocks by locking rows in a certain order in all your transactions. For example, always lock the rows in primary key order.

Finally, if you don't get many deadlocks, you don't have to try and avoid them. All that is required is to repeat the transaction if you get a deadlock. This works fine, but causes performance problems if you have too many deadlocks.

Sorry, but with the little data you provide it is not possible to say more. You will probably have to look at all the statements issued by the transactions involved in the deadlock.

user_64643080

So in one place, you update Score and someplace else they are deleted?

I think the culprit is where you iterate over list_of_scores_to_update. What you need to do instead is prefetch batches/paginate the queryset and then open transaction for the update of this batch, not the entire queryset. First select relevant pks without the lock and then update a subset of them within a transaction. See iterator.

As for the delete operation, if your use case allows your could add pk to another queue. Then another task can prefetch multiple ids from this queue and try to delete. On failure, retry. That'd be a very simple solution for it to retry without a human intervention.

To sum up, I'm sure the lock for list_of_scores_to_update is too big so try to make it smaller. As per the second snippet, decouple the removal from the filtering so it's easier to retry the removal process.

user_58334290
Related