How can a Jackson converter be written to handle generic collection types?

OrderedSetConverter does not know which type must be instantiated during deserialisation process. We need to instruct deserialisation process which type to use. If you want to keep converter generic you need to provide this information when converter is created. To extend default behaviour you need to implement custom com.fasterxml.jackson.databind.cfg.HandlerInstantiator:

class ConverterHandlerInstantiator extends HandlerInstantiator {

    @Override
    public Converter<?, ?> converterInstance(MapperConfig<?> config, Annotated annotated, Class<?> implClass) {
        if (config instanceof DeserializationConfig) {
            JsonDeserialize jsonDeserialize = annotated.getAnnotation(JsonDeserialize.class);
            Class contentAs = jsonDeserialize.contentAs();
            if (contentAs != Void.class) {
                return new OrderedSetConverter<>(contentAs);
            }
        }
        return super.converterInstance(config, annotated, implClass);
    }

    @Override
    public JsonDeserializer<?> deserializerInstance(DeserializationConfig config, Annotated annotated, Class<?> deserClass) {
        return null;
    }

    @Override
    public KeyDeserializer keyDeserializerInstance(DeserializationConfig config, Annotated annotated, Class<?> keyDeserClass) {
        return null;
    }

    @Override
    public JsonSerializer<?> serializerInstance(SerializationConfig config, Annotated annotated, Class<?> serClass) {
        return null;
    }

    @Override
    public TypeResolverBuilder<?> typeResolverBuilderInstance(MapperConfig<?> config, Annotated annotated, Class<?> builderClass) {
        return null;
    }

    @Override
    public TypeIdResolver typeIdResolverInstance(MapperConfig<?> config, Annotated annotated, Class<?> resolverClass) {
        return null;
    }
}

As you noticed above code uses contentAs property from JsonDeserialize annotation. We need to specify it in your class:

class MyType {
    @JsonSerialize(converter = OrderedSetConverter.class)
    @JsonDeserialize(converter = OrderedSetConverter.class, contentAs = DayOfWeek.class)
    private Set<DayOfWeek> myValues;
}

This value is not needed when we serialise objects, so, we can skip it. Now, we need to register our custom instantiator:

ObjectMapper mapper = new ObjectMapper();
mapper.setHandlerInstantiator(new ConverterHandlerInstantiator());

At the end, our generic converter:

class OrderedSetConverter<E extends Comparable<? super E>> extends StdConverter<Set<E>, Set<E>> {
    private final Class<E> contentClass;

    OrderedSetConverter() {
        this(null);
    }

    public OrderedSetConverter(Class<E> contentClass) {
        this.contentClass = contentClass;
    }

    @Override
    public Set<E> convert(Set<E> value) {
        return value == null ? null : value.stream()
                .sorted(Comparator.nullsLast(Comparator.naturalOrder()))
                .collect(Collectors.toCollection(LinkedHashSet::new));
    }

    @Override
    public JavaType getInputType(TypeFactory typeFactory) {
        if (contentClass == null) {
            return super.getInputType(typeFactory);
        }
        return typeFactory.constructCollectionType(Set.class, contentClass);
    }
}

Another answer pointed out that a custom Jackson HandlerInstantiator could be used to set a type variable within the Converter at object construction time. That answer required manually specifying the collection type using @JsonDeserialize.contentAs. After experimenting with this approach, I found a way to automatically deduce the type variable within the HandlerInstantiator:

HandlerInstantiator

public class OrderedSetConverterHandlerInstantiator extends HandlerInstantiator {
    @Override
    public Converter<?, ?> converterInstance(
            MapperConfig<?> config, Annotated annotated, Class<?> implClass) {
        if (OrderedSetJacksonConverter.class.equals(implClass)) {
            return new OrderedSetJacksonConverter<>(
                    getSetContentType(config, annotated));
        } else {
            return null;
        }
    }

    private JavaType getSetContentType(MapperConfig<?> config, Annotated annotated) {
        Type setType = getPropertyType(config, annotated);
        return TypeFactory.defaultInstance().constructType(setType).getContentType();
    }

    private Type getPropertyType(MapperConfig<?> config, Annotated annotated) {
        // TODO: Enhance to handle other cases, such as annotated constructor parameter
        BeanPropertyDefinition beanPropertyDefinition =
                SimpleBeanPropertyDefinition.construct(config, 
                        (AnnotatedMember) annotated);
        AnnotatedMember setter = beanPropertyDefinition.getSetter();
        AnnotatedMember getter = beanPropertyDefinition.getGetter();
        AnnotatedMember field = beanPropertyDefinition.getField();

        Type setType;
        if (setter != null) {
            setType = ((Method) setter.getMember()).getGenericParameterTypes()[0];
        } else if (getter != null) {
            setType = ((Method) getter.getMember()).getGenericReturnType();
        } else if (field != null) {
            setType = ((Field) field.getMember()).getGenericType();
        } else {
            throw new UnsupportedOperationException(
                    "Annotated type not yet supported: " + annotated.getClass());
        }
        return setType;
    }
    
    // Implement other required methods such as serializerInstance to return null
}

Converter

public class OrderedSetConverter<T extends Comparable<? super T>> 
        extends StdConverter<Set<T>, Set<T>> {
    private final JavaType contentType;

    public OrderedSetJacksonConverter(JavaType contentType) {
        this.contentType = contentType;
    }

    @Override
    public Set<T> convert(Set<T> value) {
        return value == null ? null : value.stream()
                .sorted(Comparator.nullsLast(Comparator.naturalOrder()))
                .collect(Collectors.toCollection(LinkedHashSet::new));
    }

    @Override
    public JavaType getInputType(TypeFactory typeFactory) {
        return typeFactory.constructCollectionType(Set.class, contentType);
    }

    @Override
    public JavaType getOutputType(TypeFactory typeFactory) {
        return typeFactory.constructCollectionType(Set.class, contentType);
    }
}

ObjectMapper configuration

JsonMapper jsonMapper = JsonMapper.builder()
            .handlerInstantiator(new OrderedSetConverterHandlerInstantiator())

One clunky thing about the solution is the way the property type is being determined, as it's manually iterating over all the possible annotated element types the Annotated value might be (getter/setter/field) to determine the element type. Hopefully there's a cleaner, more idiomatic approach, though if there is I haven't found it yet.

Registering the converter in the Spring framework

Spring Web ships with its own HandlerInstantiator, SpringHandlerInstantiator, which is automatically applied to the ObjectMapper created by Jackson2ObjectMapperBuilder. This handler uses any relevant beans in the classpath for the types supplied by the HandlerInstantiator.

In order to retain this behavior while still using the custom type, a HandlerInstantiator class could be written which uses the custom converter when appropriate, otherwise fall back to the default SpringHandlerInstantiator. One way to implement this is as follows:

public class DelegatingHandlerInstantiator extends SpringHandlerInstantiator {
    private final OrderedSetConverterHandlerInstantiator handler =
            new OrderedSetConverterHandlerInstantiator();

    public DelegatingHandlerInstantiator(AutowireCapableBeanFactory beanFactory) {
        super(beanFactory);
    }

    @Override
    public Converter<?, ?> converterInstance(
            MapperConfig<?> config, Annotated annotated, Class<?> implClass) {
        Converter<?, ?> converter =
                handler.converterInstance(config, annotated, implClass);
        return Objects.requireNonNullElseGet(converter,
                () -> super.converterInstance(config, annotated, implClass));
    }
}

The Jackson2ObjectMapperBuilder used by the application should then be configured to use this handler. If the application is a Spring Boot application, a Jackson2ObjectMapperBuilderCustomizer bean can be created to set the HandlerInstantiator in the Jackson2ObjectMapperBuilder bean.

@Configuration
public class JacksonConfig {
    @Bean
    public Jackson2ObjectMapperBuilderCustomizer jacksonCustomizer(
            AutowireCapableBeanFactory beanFactory) {
        return b -> b.handlerInstantiator(
                new DelegatingHandlerInstantiator(beanFactory));
    }
}