Django System Checks, Migrations, and Tests Pass Locally, but Fail in Docker CI/CD Environment

Question

Problem

Honestly, I'm just firing this one into the dark because I have exhausted every possible avenue before asking this. I'm not sure what the problem is here.

I have a Django application that works completely fine to run locally. I can run migrations. I have developed at length with this locally, and not had a single issue with the models, the testing, or any feature.

The issue here is that the second I use GitLab's CI/CD Runner and perform the exact same steps I'm performing locally I get this output.

ERRORS:
piano_gym_api.LearnerEnrolledLesson.enrolled_course: (fields.E300) Field defines a relation with model 'piano_gym_api.LearnerEnrolledCourse', which is either not installed, or is abstract.
piano_gym_api.LearnerEnrolledLesson.enrolled_course: (fields.E307) The field piano_gym_api.LearnerEnrolledLesson.enrolled_course was declared with a lazy reference to 'piano_gym_api.learnerenrolledcourse', but app 'piano_gym_api' doesn't provide model 'learnerenrolledcourse'.
piano_gym_api.LearnerEnrolledLesson.enrolled_school: (fields.E300) Field defines a relation with model 'piano_gym_api.LearnerEnrolledSchool', which is either not installed, or is abstract.
piano_gym_api.LearnerEnrolledLesson.enrolled_school: (fields.E307) The field piano_gym_api.LearnerEnrolledLesson.enrolled_school was declared with a lazy reference to 'piano_gym_api.learnerenrolledschool', but app 'piano_gym_api' doesn't provide model 'learnerenrolledschool'.

Environment

I'm using Python 3.7 with Django 2.2. My dependencies look like this:

certifi==2019.3.9
chardet==3.0.4
coreapi==2.3.3
coreschema==0.0.4
Django==2.2
django-cors-headers==3.0.2
django-extensions==2.1.7
djangorestframework==3.9.4
djangorestframework-jwt==1.11.0
gunicorn==19.9.0
idna==2.8
itypes==1.1.0
Jinja2==2.10.1
lxml==4.3.3
MarkupSafe==1.1.1
music21==5.5.0
PyJWT==1.7.1
pytz==2019.1
requests==2.22.0
six==1.12.0
sqlparse==0.3.0
uritemplate==3.0.0
urllib3==1.25.3
whitenoise==4.1.2

I'm using the free version of GitLab, with GitLab Runner.

It's a simple django project. There is one project, and one app.

My settings.conf's INSTALLED_APPS looks like this

# Application definition
INSTALLED_APPS = [
    # Django Default
    "django.contrib.admin",
    "django.contrib.auth",
    "django.contrib.contenttypes",
    "django.contrib.sessions",
    "django.contrib.messages",
    "django.contrib.staticfiles",
    # Third-Party Apps
    "corsheaders",
    "django_extensions",
    "rest_framework",
    "rest_framework.authtoken",
    "whitenoise.runserver_nostatic",
    # Custom Apps
    "piano_gym_api",
]

Steps to Run Locally

This will pass

pip3 install virtualenv
virtualenv -p python3 venv
source venv/bin/activate
pip3 install -r requirements.txt
python3 manage.py makemigrations piano_gym_api
python3 manage.py migrate
python3 manage.py test

Steps to Run in GitLab CI/CD

This will fail

I've installed the GitLab Runner

I created a .gitlab-ci.yml file in my root directory. All it has is this:

stages:
  - test

api-test:
  stage: test
  image: python:3.7
  script:
    - cd piano_gym_back_end
    # Create environment for python
    - pip3 install virtualenv
    - virtualenv -p python3 venv
    - source venv/bin/activate
    - pip3 install -r requirements.txt
    # Set up and run tests
    - python3 manage.py makemigrations piano_gym_api
    - python3 manage.py migrate
    - python3 manage.py test

Then I commit everything on the branch and run gitlab-runner exec docker api-test

Which then goes through everything and outputs this

$ python3 manage.py makemigrations piano_gym_api
SystemCheckError: System check identified some issues:

ERRORS:
piano_gym_api.LearnerEnrolledLesson.enrolled_course: (fields.E300) Field defines a relation with model 'piano_gym_api.LearnerEnrolledCourse', which is either not installed, or is abstract.
piano_gym_api.LearnerEnrolledLesson.enrolled_course: (fields.E307) The field piano_gym_api.LearnerEnrolledLesson.enrolled_course was declared with a lazy reference to 'piano_gym_api.learnerenrolledcourse', but app 'piano_gym_api' doesn't provide model 'learnerenrolledcourse'.
piano_gym_api.LearnerEnrolledLesson.enrolled_school: (fields.E300) Field defines a relation with model 'piano_gym_api.LearnerEnrolledSchool', which is either not installed, or is abstract.
piano_gym_api.LearnerEnrolledLesson.enrolled_school: (fields.E307) The field piano_gym_api.LearnerEnrolledLesson.enrolled_school was declared with a lazy reference to 'piano_gym_api.learnerenrolledschool', but app 'piano_gym_api' doesn't provide model 'learnerenrolledschool'.
ERROR: Job failed: exit code 1
FATAL: exit code 1

Models

Now I understand that this is stating that it can't find the models in the app piano_gym_api. But that doesn't make sense.

The model here is:

class LearnerEnrolledLesson(Model):
    is_enrolled = BooleanField(default=True)
    learner = ForeignKey("piano_gym_api.Learner", on_delete=CASCADE)
    # ---
    enrolled_school = ForeignKey("piano_gym_api.LearnerEnrolledSchool", on_delete=CASCADE)
    enrolled_course = ForeignKey("piano_gym_api.LearnerEnrolledCourse", on_delete=CASCADE)
    # ---
    school = ForeignKey(School, on_delete=CASCADE)
    course = ForeignKey(SchoolCourse, on_delete=CASCADE)
    lesson = ForeignKey(SchoolLesson, on_delete=CASCADE)
    order = IntegerField(default=1)

    REQUIRED_FIELDS = ["learner", "school", "course", "lesson", "enrolled_school", "enrolled_course"]

    objects = LearnerEnrolledLessonManager()

    class Meta:
        ordering = ("order",)
        unique_together = ("learner", "school", "course", "lesson", "enrolled_school", "enrolled_course", "order")

The only thing I'm doing here is using strings to reference the piano_gym_api.LearnerEnrolledSchool and piano_gym_api.LearnerEnrolledCourse.

This is only done because those models have a function that returns a LearnerEnrolledLesson and that's a circular dependency, so I have to reference the models without using the import path.

Appeal For Help

I have no idea why this is failing in my CI/CD docker environment. I'm not doing anything different. My settings.py isn't changing between the develop environment and the ci/cd environment. And, the steps are the EXACT same.

What could I possibly be doing wrong here?

user_1941720 · Answer

I found the solution for this, which was incredibly subtle.

So first and foremost, I want to thank someone named Mark ♦ from the Python Discord Channel. Thanks to their efforts sitting down with me and walking through what I worked through, we were able to diagnose this.

The issue here was subtle because in sqlite3 there were no problems. However, when I migrated to postgresql the problem arose.

The issue specifically was that I was not making it so my models were available.

Django requires models to exist in the path

django_project_name/django_app_name/models.py

or

django_project_name/django_app_name/models/

My directory structure actually looks like this

├── requirements.txt
├── manage.py
├── django_app_name
│   ├── models
│   │   └── __init__.py
│   ├── urls.py
│   └── versions
│       ├── __init__.py
│       └── v1
│           ├── __init__.py
│           ├── models
│           │   ├── __init__.py
│           │   └── ...
│           └── views
│               ├── __init__.py
│               └── ...
└── django_project_name
    └── ...

The issue here is that in order to create a versioned directory structure I had moved my models into django_project_name/django_app_name/versions/v1/

Because of this Django could not find the models because I was not explicitly making them available through the models.py file or the models directory that was expected in the Django app.

This is why I would get the error that Django couldn't find the specific models, despite them being there.

To fix this solution I had to explicitly expose them by exporting their references in the django_project_name/django_app_name/models/__init__.py file.

That looked something like this:

# REQUIRED!!!
# Django requires models that are being used to be exposed in this models
# directory
# Because we have opted to use the `versions` folder for storing the structure
# of our project, that means we need to surface those models explicitly here in
# order to remove the possibility of a missing model during the `makemigrations`
# and `migrate` commands
from django_app_name.versions.v1.models.example_model_one import *
from django_app_name.versions.v1.models.example_model_two import *
from django_app_name.versions.v1.models.example_model_three import *
...

And with this I was able to get my models to be available for makemigrations and migrate.

Very subtle :)