Filtering by custom UUIDField got broken with Django 3.2 upgrade

user_645169 0
Viewed 104

I have a Django project that I've recently upgraded from Django 2.2 to 3.2. In this project, I use a custom UUIDField that saves UUIDs to MySQL as char(36) in the following format: 12345678-1234-5678-1234-567812345678.

import uuid

from django.db import models


class UUIDField(models.UUIDField):
    """
    Overrides Django UUIDField to store full UUID's including dashes.
    """
    def __init__(self, verbose_name=None, **kwargs):
        super().__init__(verbose_name, **kwargs)
        self.max_length = 36

    def get_internal_type(self):
        return "CharField"

    def get_db_prep_value(self, value, connection, prepared=False):
        if value is None:
            return None
        if not isinstance(value, uuid.UUID):
            try:
                value = uuid.UUID(value)
            except AttributeError:
                raise TypeError(self.error_messages['invalid'] % {'value': value})

        if connection.features.has_native_uuid_field:
            return value
        return str(value)

After the upgrade, I noticed that searching for full UUIDs didn't work anymore. If I only provide the first part of the UUID (up to the first character after the first hyphen) it works as expected.

Python 3.6.9 (default, Mar 15 2022, 13:55:28)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from foobar.foo.models import Foo
>>>
>>> Foo.objects.all()
<QuerySet [<Foo: Foo object (34c46fe8-caf0-11ec-bdb9-482ae362a4c0)>]>
>>>
>>> Foo.objects.filter(id__icontains='34c46fe8-caf0-11ec-bdb9-482ae362a4c0')
<QuerySet []>
>>>
>>> Foo.objects.filter(id__icontains='34c46fe8-')
<QuerySet [<Foo: Foo object (34c46fe8-caf0-11ec-bdb9-482ae362a4c0)>]>
>>>
>>> Foo.objects.filter(id__icontains='34c46fe8-c')
<QuerySet []>
>>>

I've played with the UUIDField methods, but I can't seem to figure out what went wrong. Here's a link to a Gist using a simplified model where I got the above shell example from.

mysql django python
1 Answers

After a quite long pdb debugging session, I managed to find what the problem is. I was hoping to find raw SQL snippets along the way, but instead an object called WhereNode caught my attention.

> /home/milanb/temp/django_custom_uuid_field/django_playground/lib/python3.6/site-packages/django/db/models/sql/query.py(1399)build_filter()
-> return clause, used_joins if not require_outer else ()
(Pdb) n
--Return--
> /home/milanb/temp/django_custom_uuid_field/django_playground/lib/python3.6/site-packages/django/db/models/sql/query.py(1399)build_filter()->(<WhereNode: (...f9c4f880be0>)>, {'foo_foo'})
-> return clause, used_joins if not require_outer else ()
(Pdb) retval
(<WhereNode: (AND: <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>)>, {'foo_foo'})
(Pdb) pp locals()
{'__return__': (<WhereNode: (AND: <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>)>,
                {'foo_foo'}),
 'alias': 'foo_foo',
 'allow_joins': True,
 'allow_many': True,
 'arg': 'id__icontains',
 'branch_negated': False,
 'can_reuse': {'foo_foo'},
 'check_filterable': True,
 'clause': <WhereNode: (AND: <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>)>,
 'col': Col(foo_foo, foo.Foo.id),
 'condition': <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>,
 'current_negated': False,
 'filter_expr': ('id__icontains', '34c46fe8-caf0-11ec-bdb9-482ae362a4c0'),
 'join_info': JoinInfo(final_field=<foobar.foo.fields.UUIDField: id>, targets=(<foobar.foo.fields.UUIDField: id>,), opts=<Options for Foo>, joins=['foo_foo'], path=[], transform_function=<function Query.setup_joins.<locals>.final_transformer at 0x7f9c4f89ca60>),
 'join_list': ['foo_foo'],
 'lookup_type': 'icontains',
 'lookups': ['icontains'],
 'opts': <Options for Foo>,
 'parts': ['id'],
 'pre_joins': {},
 'reffed_expression': False,
 'require_outer': False,
 'reuse_with_filtered_relation': False,
 'self': <django.db.models.sql.query.Query object at 0x7f9c4f902c18>,
 'split_subq': True,
 'targets': (<foobar.foo.fields.UUIDField: id>,),
 'used_joins': {'foo_foo'},
 'value': '34c46fe8-caf0-11ec-bdb9-482ae362a4c0'}
(Pdb) pp clause
<WhereNode: (AND: <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>)>

Here I noticed this particular line:

 'clause': <WhereNode: (AND: <django.db.models.lookups.UUIDIContains object at 0x7f9c4f880be0>)>,

I didn't know what a django.db.models.lookups.UUIDIContains object was, so went straight to Django's source code to find out. This is en empty class made of two mixins IContains and UUIDTextMixin. The latter was the culprit in my case as it removed all hyphens from the lookup value. This was introduced as a fix after version 2.2, which I used before the upgrade, in order to support database backends without a native UUID type (e.g. MySQL 5.7 in my case).

The fix was very easy, I just had to register a custom UUIDIContains class without the UUIDTextMixin for my custom UUIDField.

My new fields.py:

import uuid

from django.db import models
from django.db.models.lookups import IContains


class UUIDField(models.UUIDField):
    """
    Overrides Django UUIDField to store full UUID's including dashes.
    """
    def __init__(self, verbose_name=None, **kwargs):
        super().__init__(verbose_name, **kwargs)
        self.max_length = 36

    def get_internal_type(self):
        return "CharField"

    def get_db_prep_value(self, value, connection, prepared=False):
        if value is None:
            return None
        if not isinstance(value, uuid.UUID):
            try:
                value = uuid.UUID(value)
            except AttributeError:
                raise TypeError(self.error_messages['invalid'] % {'value': value})

        if connection.features.has_native_uuid_field:
            return value
        return str(value)


@UUIDField.register_lookup
class UUIDIContains(IContains):
    pass
user_6451690
Related