I read a lot about best practices applying DataBase Migrations for Kubernetes Apps. Actually there are three common solutions:
- Separate CI/CD stage for DB Migrations (usually we run it before deploy new App version but in some cases we can deploy first, it doesn't matter for current question). Just decouple app and migrations, and run them in different stages one by one.
- InitContainer: we can run DB migraitions before main app container starts. It is a good solution, but we have to be careful to tune it to run migrations once for all pods and replicas, do not run on pod restarts and check probes (kubelet can kill container on timeout while migration still running)
- Separate Kubernetes Job or Helm Hook. Is close to initContainer, cannot share data with app container (but it is not really necessary for DB Migrations, so it's ok). Must be careful with timeout too - Helm can kill Job before migration is completed.
But the question - how to apply rollbacks for these solutions???
Lets try to form my ideas:
- Separate stage in CI/CD: we can save previous migration name, and then rollback to it in another stage. Pipeline: Migrations -> Deploy -> Test -> Rollback DB and ReDeploy
- But for InitContainer and HelmHook I have to idea how to realise Rollback! Do we need additional containers for that? Does helm rollback affect DB too (don't think so). What are the best practices for that?
I will be very glad to any suggestions!