How to Introduce a Secondary Tracer into JavaAgent?

[FWinkler79]

I would like to introduce a second Tracer (or SdkTracer) alongside the SDK's default.
The second Tracer shall use a different sampling strategy than the one the default one uses but shall otherwise be called whenever the SDK's default one is also called. The reason is that I need to create two traces, one of technical nature - where the default provided by the SDK is perfect and should not be impacted when the sampling strategy and behavior of the other tracer might change - and a second one where a different sampling strategy shall be used and potentially some span aggregations might be performed as well (using a second non-default exporter).

The second tracer should be called whenever the default one of the SDK is called, i.e. I don't want to introduce the new tracer to all the auto-instrumentations that are already provided by the community, but would ideally like create a decorator / wrapper around the default SdkTracer which under the hood creates the secondary Tracer instance and delegates all calls to both the default and the secondary one.

To return the wrapper, I would have to sub-class SdkTracerProvider which is what the agent uses when bootstrapping the SDK instance (available via GlobalOpenTelemetry.get(). Problem is: SdkTracerProvider is final, and so I cannot create an implementation that would return my intended wrapper. I also checked the auto-instrumentation customizers, but the one for the TracerProvider expects an SdkTracerProvider instance (rather than a TracerProvider interface instance) in return - so I see no chance of returning a TracerProvider implementation that would return the wrapped SdkTracer.

So, I checked how to modify the behavior of the io.opentelemetry.sdk.trace.SdkTracerProvider class that is instantiated by the auto-configuration within the JavaAgent. I thought of doing that via bytecode modification by adding an Advice.
I have written an extension that tries to add an Advice to the io.opentelemetry.sdk.trace.SdkTracerProvider's get(String instrumentation) method. But the advice code is never called, and it seems the SdkTracerProvider is never called.
I verified that my Advice code is correct by explicitly creating an instance of the SdkTracerProvider in my app and calling its get(...) method. But for some reason, my Advice is not called when the SdkTracerProvider is used by the JavaAgent.

I read about the JavaAgent shading certain classes, i.e. changing their package names, etc. to not clash with the same classes introduced / referenced by the application.

My questions are:

• What's the best way to introduce an additional Tracer next to the default one, so that it can use a different sampling strategy (i.e. potentially records more / different spans than the default one does, WITHOUT changing the default's behavior)?
• Which class & package name do I have to refer to when I want to add an Advice to the SdkTracerProvider that the JavaAgent uses under the hood and when it auto-configures the SDK that is then also made available via GlobalOpenTelemetry.get()?
• Is it even possible to modify the OpenTelemetry SDK implementations used by the Agent with an Advice or are these classes excluded from bytecode modifications?

[laurit]

But for some reason, my Advice is not called when the SdkTracerProvider is used by the JavaAgent.

Probably because agent excludes its own classes from instrumentation. See

opentelemetry-java-instrumentation/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/ignore/GlobalIgnoredTypesConfigurer.java

Line 130 in daf2a04

.ignoreClassLoader(AgentClassLoader.class.getName())

What's the best way to introduce an additional Tracer next to the default one, so that it can use a different sampling strategy (i.e. potentially records more / different spans than the default one does, WITHOUT changing the default's behavior)?

OpenTelemetry sdk is in https://github.com/open-telemetry/opentelemetry-java You might wish to ask these questions there. I suspect that it would be best not to do this and seek an alternative path.

Which class & package name do I have to refer to when I want to add an Advice to the SdkTracerProvider that the JavaAgent uses under the hood and when it auto-configures the SDK that is then also made available via GlobalOpenTelemetry.get()?

I believe it is not renamed, but you can easily verify it yourself just check under inst in agent jar.

Is it even possible to modify the OpenTelemetry SDK implementations used by the Agent with an Advice or are these classes excluded from bytecode modifications?

As noted above they are excluded from instrumentation but ignoreClassLoader can be overridden with allowClassLoader. It is up to you to find out whether this will work or break in some unexpected way. Since I don't know of anybody trying it before it could totally be that there are other factors that will prevent this from working besides the ignoreClassLoader call.