Django migration command hanging on Heroku even without actual new migration

user_4582775 0
Viewed 470

We are running our stack on Heroku. We are using Django 2.2 with a Postgres 11 DB. Our build pipeline (Github Actions) pushes to Heroku (git push https://git.heroku.com...) and immediately afterwards runs the migrations (heroku run python manage.py migrate --app heroku-app-name). All of that was working with a Postgres 9.6 database and is still working in our staging environment (Postgres 11). Now with production being on Postgres 11, the django migrate command is just stuck and doesn't produce any output, even so there are no actual migrations to apply.

The only differences between our production setup and our staging setup are a follower/slave in production attached to the master DB and "production workload".

In order to fix that deployment I have to run a

heroku pg:killall -a heroku-app-name
heroku restart -a heroku-app-name

At this point the migrations task in the build pipeline fails.

and afterwards migrations can be applied manually without problems:

heroku run python manage.py migrate --app heroku-app-name

So for some reason the migrations command is "waiting" for something, some database lock or whatever, yet I cannot put my finger on it. Especially odd for me is the fact that its also stuck where no migrations are to applied. Why would it be stuck there?

heroku postgresql django
1 Answers

We found the solution. There are actually three things coming together.

  1. We trigger a DB backup before running any migrations. We only do so in Production and not on staging, which was the reason why our staging environment had no issues while production had.
  2. A DB migration (even so it looks like there is nothing to apply) is actually running some commands (besides regular SELECT, UPDATE, INSERTS). E.g. in our case there was a CREATE EXTENSION ... IF NOT EXISTS always executed at the beginning.
  3. While it was possible with Postgres 9.6 to have a backup job running in parallel (I don't know what heroku is using under the hood, yet I assume a noraml pg_dump) the backup job on Postgres 11 (and others?) has now a more exclusiv lock on some operations. I assume that that a CREATE EXTENSION ... IF NOT EXISTS (even so the extension already exists) cannot be executed while having a backup job running in parallel.

(I am sure there are some Postgres internas missing to explain this more correctly)

As a result of these three things, the DB blocks the migrate operation, waiting for the backup job to finish. I have moved the daily backup job to a different time and reconfigured our pipeline to wait for the "pre-deploy"-backup to finish first.

user_45827750
Related