Django admin - limit choices of a many to many field to already selected objects

I have a model with a many to many field and want to limit the choices in the django admin detail view to the ones already selected and allow new ones to be added. I tried the following:

# models.py
class Document(models.Model):
    ...
    followups = models.ManyToManyField(
        Followup,
        blank=True,
    )
    ...

# admin.py
...
class DocumentAdminForm(ModelForm):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)
        self.fields["followups"].queryset = kwargs["instance"].followups.all()


class DocumentAdmin(admin.ModelAdmin):
    form = DocumentAdminForm
...

The filter works as expected for already selected objects but I get the following error when I try to add a new object or remove one: Select a valid choice. <OBJECT ID> is not one of the available choices.

Am I missing something with the form or do I need to do this in another way. I thought of limit_choice_to in the model but don't know how to get the right filter for this case.

I am using django 3.1