Can't get my edit action (only on the ManyToOne side) to remove the link between my entities

Does anybody know what's wrong in my code ?

Im using EasyAdminBundle 4 and I've added a ManyToOne relation between two entities (Label & Organizer). When I delete every Labels from an Organizer, it works, they are no more linked together.

But when I'm on the edit page of a Label and I remove the linked Organizer from the Tomselect and click on a save button, nothing changes.

My setOrganizer setter is effectively called with a null Organizer argument, and $this has no more Organizer attribute @ the end of this method, it's null, but when I dump my Label object in an "EasyCorp\Bundle\EasyAdminBundle\Event\AfterEntityUpdatedEvent" subscriber, it still has an Organizer.

On the other hand, when I select an other Organizer in my Label edition page, it works as expected. (same when adding the 1st Organizer from there).

Here is some of my code, feel free to ask more if it potentially can help us understand what is happening.

Cheers !

/**
 * @ORM\Entity(repositoryClass=LabelRepository::class)

 */
class Label extends AbstractOtherActor
{
    /**
    * @ORM\ManyToOne(targetEntity=Organizer::class, inversedBy="labels", fetch="EAGER")
    * @ORM\JoinColumn(nullable=true)
    */
    private $organizer;


    public function getOrganizer(): ?Organizer
    {
        return $this->organizer;
    }

    public function setOrganizer(?Organizer $organizer): self
    {
        $this->organizer = $organizer;
        return $this;
    }
}

class Organizer extends AbstractOtherActor
{
    public function __construct()
    {
        $this->labels = new ArrayCollection();
    }

    /**
    * @ORM\OneToMany(targetEntity=Label::class, mappedBy="organizer", fetch="EAGER")
    * @ORM\JoinColumn(nullable=true)
    */
    private $labels;

    /**
    * @return Collection<int, Label>
    */
    public function getLabels(): Collection
    {
        return $this->labels;
    }

    public function addLabel(Label $label): self
    {
        if (!$this->labels->contains($label)) {
            $this->labels[] = $label;
            $label->setOrganizer($this);
        }

        return $this;
    }

    public function removeLabel(Label $label): self
    {
        if ($this->labels->removeElement($label)) {
            // set the owning side to null (unless already changed)
            if ($label->getOrganizer() === $this) {
                $label->setOrganizer(null);
            }
        }

        return $this;
    }
}

My AssociationField from LabelCrudController.php

    AssociationField::new(
            'organizer'
        )->hideOnIndex(),

And my AssociationField from OrganizerCrudController.php

        AssociationField::new(
            'labels'
        )
            ->setFormTypeOption('by_reference', false)
            ->setFormTypeOption("multiple", "true")->hideOnIndex(),