Django using bulk_update to update all records

user_1230479 0
Viewed 2408

With 2.2 we now have the option to bulk update: https://docs.djangoproject.com/en/3.0/ref/models/querysets/#bulk-update

I have a model with say millions of rows and I want to efficiently update all the records. I am trying to use bulk_update, but which means I still have to load all the model objects in the memory, modify the field one by one and then use bulk update:

What I am doing:

def migrate_event_ip_to_property(apps, schema_editor):
    Event = apps.get_model('app' 'Event')
    events = Event.objects.all()

    for event in events:
        if event.ip:
            event.properties["$ip"] = event.ip

    Event.objects.bulk_update(events, ['properties'], 10000)

Since there are millions of records, can I avoid doing Event.objects.all() and load all objects into the memory even while using bulk_update?

django-models django
2 Answers
def migrate_event_ip_to_property(apps, schema_editor):
    Event = apps.get_model('app' 'Event')
    chunk = []
    # Use iterator to save memory
    for i, event in enumerate(Event.objects.only('properties', 'ip').iterator(chunk_size=10000)):
        if event.ip:
            event.properties['$ip'] = event.ip
            chunk.append(event)
        # Every 10000 events run bulk_update
        if i % 10000 == 0 and chunk:
            Event.objects.bulk_update(chunk, ['properties'])
            chunk = []
    if chunk:
        Event.objects.bulk_update(chunk, ['properties'])
user_5485620

If memory is your main concern, what about restricting the fields you load in memory with :

Event.objects.all().values('id','properties')
user_93862080
Related