Use django ORM with multiprocessing?

user_143091 0
Viewed 1148

I have a Django ORM database (mysql or sqlite), and would like to process each row with a fairly computationally intensive operation. What I have right now is something like:

entries = Entry.objects.filter(language='')
for e in entry:
    e.language = detect_language(e.text)
    e.save()

If the database was the bottleneck, I would use a transaction to speed it up. However, the detect_language function is what takes the most time. I could try to run the script multiple times in parallel, but that would introduce a race condition.

I think this could be done with multiprocessing using Pool.map() - the main process fetches the DB entries, the child processes run detect_language. I am not sure how to do it in detail, e.g. whether to save the entries in the child process or in the main process.

Is there anything to take care of when passing ORM objects between processes? Can you give a short example how to use the ORM with multiprocessing?

I've just tied it, and something like this seems to work fine. I am still wondering if there is any caveat here, or if this can be improved performance wise (e.g. batching the DB update):

def detect_and_save(obj):
    obj.language = detect_language(obj.text)
    obj.save()

with multiprocessing.Pool(processes=3) as pool:
    pool.map(detect_and_save, entries)
python-multiprocessing django-orm django python
1 Answers
  1. You don't need to pass the full ORM objects - you can just pass the parameter your function needs and save its result to the ORM objects.
  2. You could use bulk_update (available from Django 2.2 or as a 3rd party library) to save in a single query.
texts = [e.text for e in entries]
with multiprocessing.Pool() as pool:
    languages = pool.map(detect_language, texts)
for e, l in zip(entries, languages):
    e.language = l
Entry.objects.bulk_update(entries, ['language'])
user_73121750
Related