Quarkus with Microprofile OpenTracing distributed tracing doesn't work

Viewed 1022

I'm migrating Thorntail (2.4.0.Final) to Quarkus (1.11.1.Final). During test phase we have noticed that distributed tracing is not working. Tracing works (single component traces) but uber-trace-id is not passed with rest request header, so next microservice (recipient of rest request) is generating Trace from zero without span info.

In Thorntail it was done just by configuration on ClientBuilder

import org.eclipse.microprofile.opentracing.ClientTracingRegistrar;
import javax.ws.rs.client.ClientBuilder;
...
ClientBuilder clientBuilder = ClientBuilder.newBuilder();
...
ClientTracingRegistrar.configure(clientBuilder);

Comparing both I have noticed that with Thorntail there was ClientTracingRegistrarProvider available

public class ResteasyClientTracingRegistrarProvider implements ClientTracingRegistrarProvider {
    public ResteasyClientTracingRegistrarProvider() {
    }

    public ClientBuilder configure(ClientBuilder clientBuilder) {
        return this.configure(clientBuilder, Executors.newFixedThreadPool(10));
    }

    public ClientBuilder configure(ClientBuilder clientBuilder, ExecutorService executorService) {
        ResteasyClientBuilder resteasyClientBuilder = (ResteasyClientBuilder)clientBuilder;
        Tracer tracer = (Tracer)CDI.current().select(Tracer.class, new Annotation[0]).get();
        return (ClientBuilder)resteasyClientBuilder.executorService(new TracedExecutorService(executorService, tracer)).register((new Builder(tracer)).withTraceSerialization(false).build());
    }
}

with all wildfly related configuration like file META-INF/services/org.eclipse.microprofile.opentracing.ClientTracingRegistrarProvider with provider path org.wildfly.swarm.mpopentracing.deployment.ResteasyClientTracingRegistrarProvider

In Quarkus we are building rest client in the same way and there is no such provider. Is anybody aware what we need to change to have this distributed tracing feature?

Additional info: we are building rest client with jaxrs API (javax.ws.rs) and building app with dependencies:

        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-resteasy</artifactId>
        </dependency>
        <dependency>
            <groupId>io.quarkus</groupId>
            <artifactId>quarkus-smallrye-opentracing</artifactId>
        </dependency>

Thanks in advance for all suggestions and help.

1 Answers

Generally workaround looks like that: we have pre processor and post processor for rest requests which store uber-trace-id in thread local via org.jboss.logging.MDC

<dependency>
    <groupId>org.jboss.logging</groupId>
    <artifactId>jboss-logging</artifactId>
    <version>3.4.1.Final</version>
</dependency>

and then in rest client we are just using this thread local value.

Code look like that: Pre matching is reading and storing ubraTraceId if exist in incoming request header

import java.nio.charset.StandardCharsets;
import javax.annotation.Priority;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.container.PreMatching;
import javax.ws.rs.ext.Provider;
...

@Provider
@PreMatching
@Priority(1)
public class PreMatchingTraceIdRequestFilter implements ContainerRequestFilter {

...


    @Override
    public void filter(ContainerRequestContext containerRequestContext) {
        // Remove uberTraceID from the logging context which can contain a previous request.
        // Additionally done in ClearLoggingContextWriterInterceptor
        LoggingUtil.clearContext();
        String jaegerUberTraceID = decode(containerRequestContext.getHeaderString(HTTP_HEADER_UBER_TRACE_ID));
        if (nonNull(jaegerUberTraceID)) {
            LoggingUtil.setUberTraceId(jaegerUberTraceID);
            //here some logging
        }
        // else case handled by the PostMatchingTraceIdRequestFilter later in the jax-rs request processing chain.
    }

    private static String decode(String headerString) {
        return StringHelper.decode(String.valueOf(headerString), StandardCharsets.UTF_8.name());
    }
}

and in post matching we are would like to newly generated uberTraceId from Jaeger tracer and store it.

import io.jaegertracing.internal.JaegerSpanContext;
import io.opentracing.Tracer;
import javax.annotation.Priority;
import javax.inject.Inject;
import javax.ws.rs.Priorities;
import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerRequestFilter;
import javax.ws.rs.ext.Provider;
import static java.util.Objects.nonNull;

@Provider
// IMPORTANT: don't give this filter a higher priority!
// Cannot be invoked earlier because the Jaeger Client has to initialize the span for this request first.
// Tests revealed that the span is not initialized yet if this filter is given a higher priority!
@Priority(Priorities.USER)
public class PostMatchingTraceIdRequestFilter implements ContainerRequestFilter {

    private Tracer tracer;
    
    @Inject
    public void setTracer(Tracer tracer) {
        this.tracer = tracer;
    }

    @Override
    public void filter(ContainerRequestContext containerRequestContext) {
        String jaegerUberTraceId = getJaegerUberTraceID();
        if (nonNull(jaegerUberTraceId)) {
            LoggingUtil.setUberTraceId(jaegerUberTraceId);
             // here some logging
        }
        // No else since an missing active span on Tracer can happen more often than you think:
        // using @Traced(false) for example suppresses the generation of a new span
    }

    private String getJaegerUberTraceID() {
        String uberTraceId = null;
        if (nonNull(tracer) && nonNull(tracer.activeSpan())) {
            JaegerSpanContext spanContext = (JaegerSpanContext) tracer.activeSpan().context();
            uberTraceId =
                    new StringBuilder()
                            .append(spanContext.getTraceId())
                            .append(UBER_TRACE_ID_SEPARATOR)
                            .append(decode(spanContext.getSpanId()))
                            .append(UBER_TRACE_ID_SEPARATOR)
                            .append(decode(spanContext.getParentId()))
                            .append(UBER_TRACE_ID_SEPARATOR)
                            .append(spanContext.getFlags())
                            .toString();
        }
        return uberTraceId;
    }

    private static String decode(long longValue) {
        return Long.toHexString(longValue);
    }
}

LogginUtil is just wrapper for MDC from jboss logging. and it does

    public static void clearContext() {
        MDC.clear();
    }

    public static void setUberTraceId(String uberTraceId) {
        MDC.put(MDC_UBER_TRACE_ID, StringUtils.defaultIfEmpty(uberTraceId, StringUtils.EMPTY));
    }
    public static String getUberTraceId() {
        return (String) MDC.get(MDC_UBER_TRACE_ID);
    }
    public static String getUberTraceIdAndRemoveFromContext() {
        final String traceId = getUberTraceId();
        clearContext();
        return traceId;
    }

and finally rest client, here is just reading uberTraceId from thread local (via MDC)

import javax.ws.rs.client.ClientBuilder;
import org.eclipse.microprofile.opentracing.ClientTracingRegistrar;

...
    protected void addUberTraceIdHeader(Invocation.Builder builder) {
        String uberTraceId = LoggingUtil.getUberTraceId();
        if (isNotBlank(uberTraceId)) {
            builder.header(TraceExtractor.HTTP_HEADER_UBER_TRACE_ID, uberTraceId);
        } else {
            throw new IllegalStateException("Lack of uber trace id");
        }
    }
Related