Django: Prefetch_related on nested attributes

Question

I need to list all my devices. To do this I use a prefetch related to reduce the amount of queries. But one of them is consuming to much time.. I wonder if it can't go better.

I will start with the model constructions: I want a list of devices.

class Device(models.Model):
    name = models.CharField(max_length=250, null=True, blank=True)

class GatewayDevice(models.Model):
    gateway = models.ForeignKey(
        Gateway, on_delete=models.CASCADE, related_name="devices"
    )
    device = models.ForeignKey(
        Device, on_delete=models.CASCADE, related_name="gatewaydevices"
    )

In the real code the device model is larger, but that code is irrelevant. As you can see, a device has some gatewaydevices (which is a model between gateway and device)

So in my list of devices, I want for every device, the linked gateway.

This is my view:

class AdminDeviceView(GenericAPIView):
    def get_permissions(self):
        return IsAuthenticated()

    # noinspection PyMethodMayBeStatic
    def get_serializer_class(self):
        return AdminDeviceInfoSerializer

    @swagger_auto_schema(
        responses={
            200: openapi.Response(
                _("Successfully fetched all data from devices."),
                AdminDeviceInfoSerializer,
            )
        }
    )

    def get(self, request):
    """
    GET the data from all the devices.
    """

    devices = (
        Device.objects.filter()
        .all()
        .prefetch_related(
            "site__users",
            "software_update_history",
            "supplier",
            Prefetch(
                "gatewaydevices",
                queryset=GatewayDevice.objects.filter(end_date=None)
                .order_by()
                .distinct()
                .prefetch_related("gateway"),
            ),
        )
    )

    serializer_class = self.get_serializer_class()
    serializer = serializer_class(devices, many=True)
    devices_data = serializer.data

    return Response(
        {"total": devices.count(), "items": devices_data}, status=status.HTTP_200_OK
    )

This is the part of the serializer that is important:

@staticmethod
def get_gateway(device):
    return (
        GatewaySimpleSerializer(device.gatewaydevices.gateway).data
        if device.gatewaydevices.gateway
        else None
    )

I tried different approaches. This one is the current one.. Now I get this error:

AttributeError: 'RelatedManager' object has no attribute 'gateway'

user_625910 · Answer

First of all, you should be able to use .select_related("gateway") rather than .prefetch_related("gateway"). This should save you a query.

Your problem appears to be here:

if device.gatewaydevices.gateway

Here, gatewaydevices is not a single model instance, so it doesn't have the gateway property. I'm not exactly sure what your use case is, but there can be multiple gatewaydevices. Maybe you want the first (this will fail if there are zero gatewaydevices):

if device.gatewaydevices.first().gateway

Perhaps you want to know if there are any. This should work:

if device.gatewaydevices.filter(gateway__isnull=False)

This seems strange though, it feels like this code should be with the rest of the query if you want to exclude objects without gateways. I think you should just remove the if completely and always return the data. You're not saving any database time doing this.

Based on the further comment:

if device.gatewaydevices.filter(end_date__isnull=True).first()

A (possibly) better way to write the query if you're trying to avoid extra work:

gateway_devices = (
    GatewayDevice.objects
    .filter(end_date__isnull=True)
    .select_related("gateway", "device")
)

This will get everything in one query and you can just use simple attribute access to get the related objects without any extra queries, such as:

for gateway_device in gateway_devices:
   print(gateway_device.device.name)

Looking at your models (though I can't see everything I need to write this perfectly) it looks like you also want some info from Device relations, so you might need to tweak this to have device prefetch what you need, something like:

gateway_devices = (
    GatewayDevice.objects
    .filter(end_date_isnull=True)
    .select_related("gateway")
    .prefetch_related(
        Prefetch(
            "device",
            queryset=Device.objects.prefetch_related(
                # Depending how these are related you may be able to
                # save some queries using `select_related()` instead.
                "site__users", "software_update_history", "supplier"
            )
        )
    )
)

This might not be perfect, but it should give you what you need to make a good start.

Django: Prefetch_related on nested attributes

1 Answers