Use graphql-java-extended-scalars with Quarkus

I want to use ExtendedScalars.Json from the graphql-java-extended-scalars package in my application which utilizes the quarkus-smallrye-graphql however I am struggling doing so.

Given a Test model class like

public class Test {
  public String name;
  public Object description;
}

with following GraphQLApi

@GraphQLApi
public class API {
  @Inject
  SomeService someService;

  @Query
  public Test getTest() {
    return someService.getTest();
  }
}

where Test might be described by

{
  "name": "Test",
  "description": {
    "hello": "world"
  }
}

the field description should be treated as ExtendedScalars.Json, so the result of this GraphQL query

{
  test {
    description
  }
}

should be exactly

{
  "data": {
    "test": {
      "description": {
        "hello": "world"
      }
    }
  }
}

However the @ToScalar annotation does not take ExtendedScalars.Json and falling back to graphql-java with something like public GraphQLSchema.Builder addScalar(@Observes GraphQLSchema.Builder builder) { ... } to manually replace the Type and QueryType leads to a AssertException like

graphql.AssertException: All types within a GraphQL schema must have unique names. No two provided types may have the same name. No provided type may have a name which conflicts with any built in types (including Scalar and Introspection types). You have redefined the type 'Query' from being a 'GraphQLObjectType' to a 'GraphQLObjectType'

Any clue how to handle this?