feat: Add tower tracing

[jan-xyz]

Changes

This adds tracing support for the tower layer. It was addressed briefly with @cijothomas on the CNCF slack, but we didn't discuss any details of the implementation. I took the liberty for now to take 2 major design decisions:

1. General vs. Specialized tower layer: I chose to pick a general layer that does metrics and tracing together with a no-op tracer in case someone doesn't want to do tracing with this layer. I assume that long term people/teams will fully switch to OpenTelemetry for Metrics and Tracing and instead of specifying two layers which each inspect the request and response, it makes more sense to do this in one layer.

2. Backwards compatibility: I tried to leave the original types in place which have the "Metric" implementation in the name. This is maybe a more smooth migration? I am not sure how helpful that actually is, and maybe we should just break compatibility in case we want to go with a unified layer.

Note: It currently does not include context extraction.

As a follow up I would also like to provide a way to use this as a client layer, where the semantics change a bit.

Merge requirement checklist

• CONTRIBUTING guidelines followed
• Unit tests added/updated (if applicable)
• Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
• Changes in public API reviewed (if applicable)

[codecov]

Codecov Report

❌ Patch coverage is 83.06452% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.2%. Comparing base (d26f4c5) to head (d509b53).

Files with missing lines	Patch %	Lines
opentelemetry-instrumentation-tower/src/lib.rs	83.0%	42 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #431     +/-   ##
=======================================
+ Coverage   55.8%   56.2%   +0.4%     
=======================================
  Files         77      77             
  Lines      12500   12669    +169     
=======================================
+ Hits        6980    7128    +148     
- Misses      5520    5541     +21     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jan-xyz]

while working on this, I started to wonder how the overall strategy for this crate could look like, and I am wondering if my PR is maybe not going far enough. For the future we have multiple use cases for a (collection of?) tower layer(s).

So far I see at least 4:

• HTTP Server
• HTTP Client
• gRPC Server
• gRPC Client

As it stands now, it provides HTTP server support, for that it might make sense to also rename the structs accordingly to HTTPServerLayer a.s.o. I won't push the other 3 concerns into this PR, but I at least would have the plan to add these in the next couple of days/weeks if it fits this project since I need this functionality for a project I am currently working on.

It might even be an idea to provide some form of auto-detecting abstraction on top which figures out if it's in a server/client context or gRPC/HTTP exchange. I am not sure how realistic that is though, as I haven't looked into the details yet.

[cijothomas]

@cijothomas anything I can do to move this PR forward?

Sincere apologies for delay in getting back to reviewing. I left some comments in the PR. Its mostly working - the key part I want to change are

1. Avoid SDK crate dependency
2. Extract context from incoming request headers and use that context, if any, to start the span.

[francoposa]

@jan-xyz @cijothomas A full setup of this would involve linking the tracer provider to the logging-tracing bridge opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge, as the dependency libraries emit trace spans with it.

At the very least I think we would want to wire this up in the example code so people aren't missing the majority of traces the program is emitting.

For the this trace below, the child from tower-http span only appears when it is wired up with the bridge. As this PR stands, we would only get the top span.

See expanded child span from tower:

We also then get all the traces & spans from underlying libraries such as tokio, h2, etc.

Example trace:

[francoposa]

Patch to apply kitchen-sink setup for wiring the log-trace bridge the axum example is here:
axum-http-service-log-trace-bridge.txt

Side note: using Jaeger for this without anything in between to sample the traces is likely to crash your browser on those tokio traces, operations like polling will get tens of thousands of spans very quickly

[jan-xyz]

I don't think the tracing bridge should be part of the PR. This is a general problem of mixing open-telemetry and tracing and not specific to the open-telemetry-Axum instrumentation. If you want to use the bridge it's good for you but I doubt that's the general sentiment.

[francoposa]

This is a general problem of mixing open-telemetry and tracing and not specific to the open-telemetry-Axum instrumentation.

Those TRACE level logs are how the Tower ecosystem emits its trace spans, this log-tracing bridge is how we properly make the connection to the spans emitted here.

It’s a few extra lines to wire up, but that is how the Rust tracing ecosystem is for now, particularly the instrument macro. These OTel libraries enable us to correctly attach those spans from the same trace context together.

The intent of the Tower instrumentation library when I wrote it was to surface the telemetry from Tower. I doubt we want to intentionally ignore and exclude the traces and child spans created by Tower.

All it needs for now is the example code to wire it up. I am happy to add it myself.