run migrations in parallel django?

user_4597501 0
Viewed 1271

The current project has around 620 django models, which takes around 1 hour to migrate. . (uses only 12.x% of 8 core intel skylake with 1.2gigs of 30gigs ram on gcloud while migration) this metrics is combined for pg and python 2 (cpython). I assume that only a single core is used while migrating. [celery processes uses 100% CPU + 50% of ram. ]

each migration is like,

class Migration(migrations.Migration):

    initial = True

    dependencies = [
        ('dist', '0001_initial'),
    ]

    operations = [
        migrations.CreateModel(
            name='model name',
            fields=[
                ...
                ...
                60 fields
                ...
                ...
            ],
            options={
                'abstract': False,
            },
        ),
    ]

they are identical but with some minor changes to fields.

Is there any way to migrate this database faster? can I run migrations on parallel? since migrations after 30 (internal apps and dists) are not related to each other, can I run these migrations in parallel?

thank you

django-migrations parallel-processing django python
1 Answers

No, migrations are (for good reasons) designed to be applied sequentially (consider for example a scenario where one migration creates a model and another modifies it - running them in parallel could make Django try to modify a model that does not exist yet). In your case squashing the migrations could speed things up - to quote documentation:

Squashing is the act of reducing an existing set of many migrations down to one (or sometimes a few) migrations which still represent the same changes.

Django does this by taking all of your existing migrations, extracting their Operations and putting them all in sequence, and then running an optimizer over them to try and reduce the length of the list - for example, it knows that CreateModel and DeleteModel cancel each other out, and it knows that AddField can be rolled into CreateModel.

Once the operation sequence has been reduced as much as possible - the amount possible depends on how closely intertwined your models are and if you have any RunSQL or RunPython operations (which can’t be optimized through unless they are marked as elidable) - Django will then write it back out into a new set of migration files.

Read more about squashing migrations here.

user_35703520
Related