Dynamically Limit Choices of Wagtail Edit Handler

I would like to limit the choices available of a field in the Wagtail-Admin edit view. My Wagtail version is 2.16.3.

In my case, if have a page model, that describes product categories. To each category might exist some tags.

There is another page model, that describes concrete products and always is a subpage of a category page.

A product can now be described with some tags, but only tags, that belong to the product's category make sense, so I would like to restrict the edit handler of the products to these tags.

My models.py looks similar to this:

class ProductTag(ClusterableModel):
    name = models.CharField(_("Name"), max_length=50)
    
    category = ParentalKey(
        "ProductCategoryIndexPage", related_name="tags", on_delete=models.CASCADE
    )

class ProductPage(Page):

    parent_page_types = ["products.ProductCategoryIndexPage"]
 
    tags = ParentalManyToManyField("ProductTag", related_name="products")
    
    content_panels = Page.content_panels + [
        FieldPanel("tags", widget=widgets.CheckboxSelectMultiple()),
    ]
    
    
class ProductCategoryIndexPage(Page):
    subpage_types = ["products.ProductPage"]
  
    content_panels = Page.content_panels + [
        MultiFieldPanel(
            [InlinePanel("tags", label="Tags")],
            heading=_("Produkt Tags"),
        ),
    ]

My approach was to create a custom edit handler and inject the wigdet overriding the on_instance_bound method using the correct choices argument for the widget.

class ProductTagEditPanel(FieldPanel):
    def on_instance_bound(self):
        self.widget = widgets.CheckboxSelectMultiple(
            # of_product is a custom manager method, returning only the tags, that belong to the product's category
            choices=ProductTag.objects.of_product(self.instance) 
        )
        return super().on_instance_bound()

But somewhere in the form creation process, the choices are overridden again and I cannot find a good location, where to inject the custom query.

Also looking through the wagtail issues I could only find remotely related stuff and I'm wondering, if my plans are to exotic or I'm sitting to long in front of this problem..