Check if record exists when bulk POST'ing with Django REST Framework

Question

I have a list dictionaries which I've parsed to JSON with json.dumps(). I would now like to POST this data to my database using Django REST framework.

# Example Data to POST
[
   {
      "key_1":"data_1",
      "key_2":"data_2",

   },
   {
      "key_1":"data_1",
      "key_2":"data_2",
   },
   {
      "key_1":"data_3",
      "key_2":"data_4",
   }
]

If we imagine that all entries are unique (which isn't the case with the above example dataset), we can successfully batch POST this data with:

# models.py
class data(models.Model):
    key_1 = models.CharField(max_length=64, blank=True, null=True)
    key_2 = models.CharField(max_length=64, blank=True, null=True)

    class Meta:
        unique_together = (( "key_1", "key_2"))

# serializers.py
class dataSerializer(serializers.ModelSerializer):
    class Meta:
        model = data
        fields = '__all__'

# views.py
class dataViewSet(viewsets.ModelViewSet):
    queryset=data.objects.all()
    serializer_class=dataSerializer
    filter_backends=[DjangoFilterBackend]
    filterset_fields=['key_1', 'key_2']

    def create(self, request, *args, **kwargs):
        serializer = self.get_serializer(data=request.data, many=isinstance(request.data,list))
        serializer.is_valid(raise_exception=True)
        self.perform_create(serializer)
        headers = self.get_success_headers(serializer.data)
        return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)

# Initiating the POST request
api_url="localhost:8000/app/api/"
requests.post(
        f"{api_url}data/", 
        data=my_json_serialised_data, 
        headers=headers
    )

However, this will fail if some records already exist in the database ("fields must be unique together").

As per the example data, entries in the list will occasionally already be present in the database and I would therefore like to avoid POST'ing duplicates (based on the combination of fields in the model; I have specified unique_together to be explicit about which fields).

What's the best DRF way of checking whether a record exists and--if it does--to skip it, when you're dealing with a bulk POST? Should I use viewsets.ModelViewSet and override the create() method within that class? Or is there a better approach? I note that there are several ways to create records in DRF, so I'm seeking clarity on the best approach when we're dealing with a batch of data like in the above example.

user_6759844 · Answer

To solve this, you can use a custom list serializer to be able to get the validated data as a list, which will then support creating your data objects in bulk using bulk_create.

In addition, you can ignore the constraint errors (like unique constraints in your case) with bulk_create's ignore_conflicts:

On databases that support it (all but Oracle), setting the ignore_conflicts parameter to True tells the database to ignore failure to insert any rows that fail constraints such as duplicate unique values. Enabling this parameter disables setting the primary key on each model instance (if the database normally supports it).

So all in all you can do something like:

class dataListSerializer(serializers.ListSerializer)
    def create(self, validated_data):
        return data.objects.bulk_create([
            data(**validated) for validated in validated_data
        ], ignore_conflicts=True)


class dataSerializer(serializers.ModelSerializer):
    class Meta:
        model = data
        fields = '__all__'
        list_serializer_class = dataListSerializer

When many=True is flagged in the serializer, dataListSerializer's create will be used instead of dataSerializer which will support the abovementioned create in bulk.

Check if record exists when bulk POST'ing with Django REST Framework

1 Answers