Clear ReferenceArrayInput selection on filter change

I have 2 ReferenceArrayInputs, one of which is wrapped in a FormDataConsumer.

The first one is for category, the second for sub-category. The second one depends on the first one so I conditionally render it.

When I select a value for category, then select a value for subcategory, and finally change the category value again I still see my selection in the subcategory.

<ArrayInput source={`shops.${shopIndex}.categories`}>
  <SimpleFormIterator disableReordering>
    <ReferenceInput source="categoryId" reference="categories">
      <SelectInput optionText="name" />
    </ReferenceInput>
    <FormDataConsumer>
      {({ formData, getSource, scopedFormData, ...rest }) => {
        // No category, no render. I can't filter subcategories.
        if (!scopedFormData.categoryId) {
          return null;
        }

        return (
          <SelectArrayInput
            source={getSource('subCategoryIds')}
            reference="subCategories"
            perPage={500}
            filter={{ categories: [scopedFormData.categoryId] }}
          >
            <ReferenceArrayInput optionText="name" />
          </SelectArrayInput>
        );
      }}
    </FormDataConsumer>
  </SimpleFormIterator>
</ArrayInput>

Filtering works great, toggling visibility looks great... I just want to clear the subcategory value when changing the category.

I already tried useFormContext() with a setValue and resetValue, but neither worked. I'm not sure if this is a caching issue or simply because it's an array.

Regarding versions, this is 4.1.0 all the way up to the master branch (tried all of them).