How to configure SpringBoot ObjectMapper

I need to configure ObjectMapper in two places. I have a jar which is a dependency of my other projects. So I have something like this

@Primary
@Bean
public ObjectMapper objectMapper() {
    final ObjectMapper mapper = new ObjectMapper();
    <customize stuff>
   
    return mapper;
}

In one of my other projects, I need to make some additional customizations on top of the ones above, but if I do the same thing, it creates a new ObjectMapper instead of making the changes to the one I want all my projects using. How can I make changes to the one and only Spring ObjectMapper

Update

I was able to get this working better by implementing Jackson2ObjectMapperBuilderCustomizer. But I have another, probably related, issue that in that the child project also uses Yaml. So I have a bean

@Bean("yamlMapper")
public YAMLMapper yamlObjectMapper() {
    return new YAMLMapper();
}

This works fine as long as I don't have the Jackson2ObjectMapperBuilderCustomizer. But if I do, it seems to want to use Yaml for both the input and output. My input and output is still Json. But it uses Yaml for some internal files. But I get errors because it's trying to use Yaml to read the input to the service. This only happens if I try to customize the ObjectMapper with Jackson2ObjectMapperBuilderCustomizer