Using south to refactor a Django model with inheritance

Question

I was wondering if the following migration is possible with Django south and still retain data.

Before:

I currently have two apps, one called tv, one called movies, each with a VideoFile model (simplified here):

tv/models.py:

class VideoFile(models.Model):
    show = models.ForeignKey(Show, blank=True, null=True)
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

movies/models.py:

class VideoFile(models.Model):
    movie = models.ForeignKey(Movie, blank=True, null=True)
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

After:

Because the two videofile objects are so similar I want to get rid of duplication and create a new model in a separate app called media that contains a generic VideoFile class and use inheritance to extend it:

media/models.py:

class VideoFile(models.Model):
    name = models.CharField(max_length=1024, blank=True)
    size = models.IntegerField(blank=True, null=True)
    ctime = models.DateTimeField(blank=True, null=True)

tv/models.py:

class VideoFile(media.models.VideoFile):
    show = models.ForeignKey(Show, blank=True, null=True)

movies/models.py:

class VideoFile(media.models.VideoFile):
    movie = models.ForeignKey(Movie, blank=True, null=True)

So my question is, how can I accomplish this with django-south and still maintain existing data?

All three these apps are already managed by south migrations and according to the south documentation it is bad practice to combine a schema and data migration and they recommend it should be done in a few steps.

I think it could be done using separate migrations like this (assuming media.VideoFile is already created)

Schema migration to rename all fields in tv.VideoFile and movies.VideoFile that will move to the new media.VideoFile model, maybe to something like old_name, old_size, etc
Schema migration to tv.VideoFile and movies.VideoFile to inherit from media.VideoFile
Data migration to copy old_name to name, old_size to size, etc
Scheme migration to remove old_ fields

Before I go through all that work, do you think that will work? Is there a better way?

If you're interested, the project is hosted here: http://code.google.com/p/medianav/

user_346587 · Answer

I did try to walk through the solution outlined by T Stone and while I think it's a superb starter and explains how things should be done I ran into a few problems.

I think mostly you don't need to create the table entry for the parent class anymore, i.e. you don't need

new_movie.videofile_ptr = orm['media.VideoFile'].objects.create()

anymore. Django will now do this automatically for you (if you have non-null fields then the above did not work for me and gave me a database error).

I think it is probably due to changes in django and south, here is a version that worked for me on ubuntu 10.10 with django 1.2.3 and south 0.7.1. The models are a little different, but you will get the gist:

Initial setup

post1/models.py:

class Author(models.Model):
    first = models.CharField(max_length=30)
    last = models.CharField(max_length=30)

class Tag(models.Model):
    name = models.CharField(max_length=30, primary_key=True)

class Post(models.Model):
    created_on = models.DateTimeField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)
    title = models.CharField(max_length=128, blank=True)
    content = models.TextField(blank=True)

post2/models.py:

class Author(models.Model):
    first = models.CharField(max_length=30)
    middle = models.CharField(max_length=30)
    last = models.CharField(max_length=30)

class Tag(models.Model):
    name = models.CharField(max_length=30)

class Category(models.Model):
    name = models.CharField(max_length=30)

class Post(models.Model):
    created_on = models.DateTimeField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)
    title = models.CharField(max_length=128, blank=True)
    content = models.TextField(blank=True)
    extra_content = models.TextField(blank=True)
    category = models.ForeignKey(Category)

There is obviously a lot of overlap, so I wanted to factor the commonalities out into a general post model and only keep the differences in the other model classes.

new setup:

genpost/models.py:

class Author(models.Model):
    first = models.CharField(max_length=30)
    middle = models.CharField(max_length=30, blank=True)
    last = models.CharField(max_length=30)

class Tag(models.Model):
    name = models.CharField(max_length=30, primary_key=True)

class Post(models.Model):
    created_on = models.DateTimeField()
    author = models.ForeignKey(Author)
    tags = models.ManyToManyField(Tag)
    title = models.CharField(max_length=128, blank=True)
    content = models.TextField(blank=True)

post1/models.py:

import genpost.models as gp

class SimplePost(gp.Post):
    class Meta:
        proxy = True

post2/models.py:

import genpost.models as gp

class Category(models.Model):
    name = models.CharField(max_length=30)

class ExtPost(gp.Post):
    extra_content = models.TextField(blank=True)
    category = models.ForeignKey(Category)

If you want to follow along you will first need to get these models into south:

$./manage.py schemamigration post1 --initial
$./manage.py schemamigration post2 --initial
$./manage.py migrate

Migrating the data

How to go about it? First write the new app genpost and do the initial migrations with south:

$./manage.py schemamigration genpost --initial

(I am using $ to represent the shells prompt, so don't type that.)

Next create the new classes SimplePost and ExtPost in post1/models.py and post2/models.py respectively (don't delete the rest of the classes yet). Then create schemamigrations for these two as well:

$./manage.py schemamigration post1 --auto
$./manage.py schemamigration post2 --auto

Now we can apply all these migrations:

$./manage.py migrate

Let's get to the heart of the matter, migrating the data from post1 and post2 to genpost:

$./manage.py datamigration genpost post1_and_post2_to_genpost --freeze post1 --freeze post2

Then edit genpost/migrations/0002_post1_and_post2_to_genpost.py:

class Migration(DataMigration):

    def forwards(self, orm):

        # 
        # Migrate common data into the new genpost models
        #
        for auth1 in orm['post1.author'].objects.all():
            new_auth = orm.Author()
            new_auth.first = auth1.first
            new_auth.last = auth1.last
            new_auth.save()

        for auth2 in orm['post2.author'].objects.all():
            new_auth = orm.Author()
            new_auth.first = auth2.first
            new_auth.middle = auth2.middle
            new_auth.last = auth2.last
            new_auth.save()

        for tag in orm['post1.tag'].objects.all():
            new_tag = orm.Tag()
            new_tag.name = tag.name
            new_tag.save()

        for tag in orm['post2.tag'].objects.all():
            new_tag = orm.Tag()
            new_tag.name = tag.name
            new_tag.save()

        for post1 in orm['post1.post'].objects.all():
            new_genpost = orm.Post()

            # Content
            new_genpost.created_on = post1.created_on
            new_genpost.title = post1.title
            new_genpost.content = post1.content

            # Foreign keys
            new_genpost.author = orm['genpost.author'].objects.filter(\
                    first=post1.author.first,last=post1.author.last)[0]

            new_genpost.save() # Needed for M2M updates
            for tag in post1.tags.all():
                new_genpost.tags.add(\
                        orm['genpost.tag'].objects.get(name=tag.name))

            new_genpost.save()
            post1.delete()

        for post2 in orm['post2.post'].objects.all():
            new_extpost = p2.ExtPost() 
            new_extpost.created_on = post2.created_on
            new_extpost.title = post2.title
            new_extpost.content = post2.content

            # Foreign keys
            new_extpost.author_id = orm['genpost.author'].objects.filter(\
                    first=post2.author.first,\
                    middle=post2.author.middle,\
                    last=post2.author.last)[0].id

            new_extpost.extra_content = post2.extra_content
            new_extpost.category_id = post2.category_id

            # M2M fields
            new_extpost.save()
            for tag in post2.tags.all():
                new_extpost.tags.add(tag.name) # name is primary key

            new_extpost.save()
            post2.delete()

        # Get rid of author and tags in post1 and post2
        orm['post1.author'].objects.all().delete()
        orm['post1.tag'].objects.all().delete()
        orm['post2.author'].objects.all().delete()
        orm['post2.tag'].objects.all().delete()


    def backwards(self, orm):
        raise RuntimeError("No backwards.")

Now apply these migrations:

$./manage.py migrate

Next you can delete the now redundant parts from post1/models.py and post2/models.py and then create schemamigrations to update the tables to the new state:

$./manage.py schemamigration post1 --auto
$./manage.py schemamigration post2 --auto
$./manage.py migrate

And that should be it! Hopefully it all works and you have refactored your models.

Using south to refactor a Django model with inheritance

Before:

After:

4 Answers

Initial setup

Migrating the data