OpenTelemetry Propagation Issue With Azure ServiceBus Queues using Go

user_695170 0
Viewed 183

I have three services which subscribe and publish to separate queues on azure service bus. I've just added open telemetry for these services and it doesn't appear to propagate the trace ID between services.

For example, a single trace would ideally include the flow between services:

Service 1 -> Queue1 -> Service 2 -> Queue2 -> Service 3

It looks like I'm supposed to Inject/Extract this information between services, but there appears to be limited documentation/examples on how to do this, because apparently it's supposed to "just work" using the W3C Trace Context?

If anyone can provide any insight or pointers into exactly what I should be doing in order to achieve the desired outcome, that would be greatly appreciated.

Many thanks.

azure-sdk azureservicebus open-telemetry go azure
1 Answers

I remember doing something similar with OpenCensus trace library when I needed to propagate traces between multiple GRPC services. I took the existing trace span, fed it to Binary function, got the bytes, set it as data on the outgoing context, then on the receiving side I extracted the trace span from the incoming context and fed it to trace.StartSpanWithRemoteParent.

This is the implementation of the util that was adding trace to context:

package grpcutil

import (
    "context"
    "encoding/base64"

    "github.com/grpc-ecosystem/go-grpc-middleware/util/metautils"
    "go.opencensus.io/trace"
    "go.opencensus.io/trace/propagation"
    "google.golang.org/grpc/metadata"
)

const TraceContextMDKey = "trace_ctx"

func NewTraceableContext(ctx context.Context, span *trace.Span) context.Context {
    return metadata.AppendToOutgoingContext(
        ctx,
        TraceContextMDKey, base64.StdEncoding.EncodeToString(propagation.Binary(span.SpanContext())),
    )
}

func GetRemoteTraceContext(ctx context.Context) (trace.SpanContext, bool) {
    var traceCtx []byte
    extCtx := metautils.ExtractIncoming(ctx).Get(TraceContextMDKey)
    if len(extCtx) > 0 {
        traceCtx, _ = base64.StdEncoding.DecodeString(extCtx)
    }
    return propagation.FromBinary(traceCtx)
}

How it looked on the receiving side:

remoteTraceCtx, _ := grpcutil.GetRemoteTraceContext(ctx)
_, span := trace.StartSpanWithRemoteParent(ctx, "name", remoteTraceCtx)
defer span.End()

Hope this points you in the right direction.

user_21449890
Related