Best practices for in-app database migration for Sqlite

I maintain an application that periodically needs to update a sqlite database and migrate old databases to the new schema and here's what I do:

For tracking the database version, I use the built in user-version variable that sqlite provides (sqlite does nothing with this variable, you are free to use it however you please). It starts at 0, and you can get/set this variable with the following sqlite statements:

> PRAGMA user_version;  
> PRAGMA user_version = 1;

When the app starts, I check the current user-version, apply any changes that are needed to bring the schema up to date, and then update the user-version. I wrap the updates in a transaction so that if anything goes wrong, the changes aren't committed.

For making schema changes, sqlite supports "ALTER TABLE" syntax for certain operations (renaming the table or adding a column). This is an easy way to update existing tables in-place. See the documentation here: http://www.sqlite.org/lang_altertable.html. For deleting columns or other changes that aren't supported by the "ALTER TABLE" syntax, I create a new table, migrate date into it, drop the old table, and rename the new table to the original name.

The answer from Just Curious is dead-on (you got my point!), and it's what we use to track the version of the database schema that is currently in the app.

To run through the migrations that need to occur to get user_version matching the app's expected schema version, we use a switch statement. Here's a cut-up example of what this look like in our app Strip:

- (void) migrateToSchemaFromVersion:(NSInteger)fromVersion toVersion:(NSInteger)toVersion { 
    // allow migrations to fall thru switch cases to do a complete run
    // start with current version + 1
    [self beginTransaction];
    switch (fromVersion + 1) {
        case 3:
            // change pin type to mode 'pin' for keyboard handling changes
            // removing types from previous schema
            sqlite3_exec(db, "DELETE FROM types;", NULL, NULL, NULL);
            NSLog(@"installing current types");
            [self loadInitialData];
        case 4:
            //adds support for recent view tracking
            sqlite3_exec(db, "ALTER TABLE entries ADD COLUMN touched_at TEXT;", NULL, NULL, NULL);
        case 5:
            {
                sqlite3_exec(db, "ALTER TABLE categories ADD COLUMN image TEXT;", NULL, NULL, NULL);
                sqlite3_exec(db, "ALTER TABLE categories ADD COLUMN entry_count INTEGER;", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS categories_id_idx ON categories(id);", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS categories_name_id ON categories(name);", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS entries_id_idx ON entries(id);", NULL, NULL, NULL);

               // etc...
            }
    }

    [self setSchemaVersion];
    [self endTransaction];
}

The best solution IMO is to build a SQLite upgrade framework. I had the same problem (in the C# world) and I built my own such framework. You can read about it here. It works perfectly and makes my (previously nightmarish) upgrades work with minimal effort on my side.

Although the library is implemented in C#, the ideas presented there should work fine in your case also.

For .net you can use lib:

EntityFrameworkCore.Sqlite.Migrations

It is simple, so for any other platform you can easily implement the same behavior as in lib.

If you change the database schema and all code that's using it in lockstep, as is likely to be the case in embedded and phone-located apps, the problem is actually well under control (nothing comparable to the nightmare that's schema migration on an enterprise DB that may be serving hundreds of apps -- not all under the DBA's control either;-).

In my article Simple declarative schema migration for SQLite we work out the schema changes automatically by creating a pristine in-memory database, and comparing the schema against your current database by querying the "sqlite_schema" tables from both. Then we follow the 12 step procedure from the SQLite documentation to safely modify the tables.

You can define the schema however you like (an ORM, or plain SQL "CREATE TABLE" statements, etc) as long as you can use it to create a new in-memory database. This means that you only have to maintain the schema in one place, and changes are applied automatically when your application starts up.

Of course there are limitations — in particular this doesn't handle data migrations, only schema migrations; and new columns must allow null or have a default value specified. But overall it's a joy to work with.