I have a Go Service which gets traced by opentelemetry. For now, we trace by ratio.
Now for error analysis we would like to change tracing to be always enforced once we call RecordError.
Initial idea was to implement a Sampler, which would look at the collected Events of a Span. From the interfaces it seems like we have no access to this.
So how could I do this instead?
The Samplers from the SDK are head based i.e sampling decision is made at the beginning of span and interfaces in OTEL SDKs are built around this requirement. One way you could work around this is to always sample in the beginning and write your own custom span processor which makes the decision to whether or not sample the span. This can get hairy quickly since it breaks the trace and some span might have parentId set to span which is dropped.
Or Another way you could achieve your goal but with the addition of new component. Do tail sampling i.e drop/keep the entire trace if it matches certain criteria and there is processor for that in opentelemetry collector contrib https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor. Please note the agent/gateway deployment of collector which is doing tail sampling has to have the access to entire trace.
