JPA Many To Many With @Where clause produces JpaObjectRetrievalFailureException

I have two entities Tasks and Labels and I'd like to join them via a @ManyToMany relation. One of the entities has a @Where clause which is automatically added to the resulting query created by hibernate. The query is generated using an @NamedEntityGraph in the corresponding JPA Repository If a reference exists in the reference table to an entiy that is filtered out by the @Where clause, Hibernate produces these warnings and finally throws an exception

• HH000100: Fail-safe cleanup (collections)
• HHH000160: On CollectionLoadContext#cleanup, localLoadingCollectionKeys contained [1] entries
• JpaObjectRetrievalFailureException <--- unable to find an entiy that was filtered out due to the query generated is using left outer joins

Entity Task

@Entity 
public class Task {

  @Id private Long Id

  @ManyToMany
  @JoinTable(name = "TASK_LABEL_REFS",
          joinColumns = {@JoinColumn(name = "TASK_ID")},
          inverseJoinColumns = {@JoinColumn(name = "LABEL_ID")})
  private Set<TaskLabel> labels = new HashSet<>();

  ...getters / setters etc.
}

Entity Label

@Entity
@Where(clause = "is_active = 1")
public class TaskLabel {

  @Id private Long Id

  @Column(nullable = false)
  private Boolean isActive = false;

  @Column(nullable = false)
  private String labelKey;

  ...getters / setters etc.
}

Entity Graph

@NamedEntityGraph(
   name = "Task.labels",
   attributeNodes = {
      @NamedAttributeNode("labels")
   })

The generated query looks like this

select
    task0_.id as id1_9_0_,
    tasklabel2_.id as id1_4_1_,
    tasklabel2_.is_active as is_active2_4_1_,
    tasklabel2_.label_key as label_key3_4_1_,
    labels1_.task_id as task_id1_3_0__,
    labels1_.label_id as label_id2_3_0__ 
from
    tasks task0_ 
left outer join
    task_label_refs labels1_ 
        on task0_.id=labels1_.task_id 
left outer join
    task_labels tasklabel2_ 
        on labels1_.label_id=tasklabel2_.id 
        and (
            tasklabel2_.is_active = 1
        ) 
where task0_.id=?

How can this be fixed?