Hibernate throws when deleting an entity which references another entity which is also contained in a set

As you can also see in the following code I have a provider (basically a company) which can have an address.

It has also a set of addresses which are related to it (imagine a number of properties, as in houses). Naturally, it's own address is also part of this set.

public class Provider {
    // ...
    @OneToOne(optional = false, cascade = {CascadeType.DETACH, CascadeType.PERSIST, CascadeType.MERGE}, orphanRemoval = true)
    @JoinColumn(name="prv_contactChannel_id", referencedColumnName = "id")
    public ContactChannel addressContactChannel;
    
    @OneToMany(mappedBy = "provider", cascade = {CascadeType.PERSIST}, orphanRemoval = true, fetch = FetchType.LAZY)
    public Set<ContactChannel> contactChannels;
    // ...

}

A ContactChannel has a reference to the provider it is associated with.

public class ContactChannel {
    // ...
    @ManyToOne(cascade=CascadeType.MERGE, fetch = FetchType.LAZY)
    @JoinColumn(name = "provider_id")
    private Provider provider;
    // ...
}

When saving a provider entity everything is fine. ContactChannels are persisted and everything. But while deleting a provider the following error occurs:

A javax.persistence.PersistenceException has been caught, org.hibernate.PropertyValueException:
not-null property references a null or transient value : models.Provider.addressContactChannel

The error doesn't occur when addressContactChannel is not made part of the set. Do any of you have an idea how to fix this?