How can a docker-compose configuration transition from using anonymous volumes to named volumes while maintaining existing data?

user_137744 0
Viewed 180

Is there a way to migrate from a docker-compose configuration using all anonymous volumes to one using named volumes without needing manual intervention to maintain data (e.g. manually copying folders)? This could entail having users run a script on the host machine but there would need to be some safeguard against a subsequent docker-compose up succeeding if the script hadn't been run.

I contribute to an open source server application that users install on a range of infrastructure. Our users are typically not very technical and are resource-constrained. We have provided a simple docker-compose-based setup. Persistent data is in a containerized postgres database which stores its data on an anonymous volume. All of our administration instructions involve stopping running containers but not bringing them down.

This works well for most users but some users have ended up doing docker-compose down either because they have a bit of Docker experience or by simple analogy to up. When they bring their server back up, they get new anonymous volumes and it looks like they have lost their data. We have provided instructions for recovering from this state but it's happening often enough that we're reconsidering our configuration and exploring transitioning to named volumes.

We have many users happily using anonymous volumes and following our administrative instructions exactly. These are our least technical users and we want to make sure that they are not negatively affected by any change we make to the docker-compose configuration. For that reason, we can't "just" change the docker-compose configuration to use named volumes and provide a script to migrate data. There's too high of a risk that users would forget/fail to run the script and end up thinking they had lost all their data. This kind of approach would be fine if we could somehow ensure that bringing the service back up with the new configuration only succeeds if the data migration has been completed.

Side note for those wondering about our choice to use a containerized database: we also have a path for users to specify an external db server (e.g. RDS) but this is only accessible to our most resourced users.

Edit: Here is a similar ServerFault question.

docker-compose
1 Answers

Given that you're using an official PostgreSQL image, you can exploit their database initialization system

If you would like to do additional initialization in an image derived from this one, add one or more *.sql, *.sql.gz, or *.sh scripts under /docker-entrypoint-initdb.d (creating the directory if necessary). After the entrypoint calls initdb to create the default postgres user and database, it will run any *.sql files, run any executable *.sh scripts, and source any non-executable *.sh scripts found in that directory to do further initialization before starting the service.

with a change of PGDATA

This optional variable can be used to define another location - like a subdirectory - for the database files. The default is /var/lib/postgresql/data. If the data volume you're using is a filesystem mountpoint (like with GCE persistent disks) or remote folder that cannot be chowned to the postgres user (like some NFS mounts), Postgres initdb recommends a subdirectory be created to contain the data.

to solve the problem. The idea is that you define a different location for Postgres files and mount a named volume there. The new location will be empty initially and that will trigger database initialization scripts. You can use this to move data from anonymous volume and do this exactly once.

I've prepared an example for you to test this out. First, create a database on an anonymous volume with some sample data in it:

docker-compose.yml:

version: "3.7"
services:
  postgres:
    image: postgres
    environment:
      POSTGRES_PASSWORD: test
    volumes:
      - ./test.sh:/docker-entrypoint-initdb.d/test.sh

test.sh:

#!/bin/bash
set -e

psql -v ON_ERROR_STOP=1 --username "postgres" --dbname "postgres" <<-EOSQL
    CREATE TABLE public.test_table (test_column integer NOT NULL);
    INSERT INTO public.test_table VALUES (1);
    INSERT INTO public.test_table VALUES (2);
    INSERT INTO public.test_table VALUES (3);
    INSERT INTO public.test_table VALUES (4);
    INSERT INTO public.test_table VALUES (5);
EOSQL

Note how this test.sh is mounted, it should be in /docker-entrypoint-initdb.d/ directory in order to be executed at the initialization stage. Bring the stack up and down to initialize the database with this sample data.

Now create a script to move the data:

move.sh:

#!/bin/bash
set -e

rm -rf $PGDATA/*
mv /var/lib/postgresql/data/* "$PGDATA/"

and update the docker-compose.yml with a named volume and a custom location for data:

docker-compose.yml:

version: "3.7"
services:
  postgres:
    image: postgres
    environment:
      POSTGRES_PASSWORD: test
      # set a different location for data
      PGDATA: /pgdata
    volumes:
      # mount the named volume
      - pgdata:/pgdata
      - ./move.sh:/docker-entrypoint-initdb.d/move.sh

volumes:
  # define a named volume
  pgdata: {}

When you bring this stack up it won't find the database (because named volume is initially empty) and Postgres will run initialization scripts. First it runs its own script to create an empty database then it runs custom scripts from the /docker-entrypoint-initdb.d directory. In this example I mounted move.sh into that directory, which will erase temporary database and move old database to the new location.

user_113445020
Related