ChatModelPromptContentObservationHandler should tag span attributes for LLM Observability, not only log to SLF4J

[AneshkaGoyal]

Please do a quick search on GitHub issues first, there might be already a duplicate issue for the one you are about to create.
If the bug is trivial, just go ahead and create the issue. Otherwise, please take a few moments and fill in the following sections:

Bug description
Spring AI ships ChatModelPromptContentObservationHandler to expose LLM prompt and completion content, but it only emits them via SLF4J logging. It does not attach them as OpenTelemetry span attributes (gen_ai.input.messages / gen_ai.output.messages).
As a result, observability backends that rely on OTEL span data — such as Datadog LLM Observability, Honeycomb, or any OTel-native platform — cannot display the actual prompt and completion content in traces. Users are forced to write and register custom ObservationHandler implementations just to bridge this gap (see workaround below).
The same gap exists for tool/function call inputs and outputs: there is no built-in handler that records these as span attributes.
Spring AI already defines the semantic conventions (gen_ai.*) and wires up the ObservationRegistry, so emitting this data as span tags should be supported out of the box.

Environment
Spring AI: 1.1.x (please confirm which version introduced ChatModelPromptContentObservationHandler)
Java: 21
Tracing: Micrometer Tracing + OpenTelemetry bridge (e.g. micrometer-tracing-bridge-otel)
Observability backend: Datadog (LLM Observability), but affects any OTEL-native backend

Steps to reproduce
Add Spring AI with a chat model (e.g. Anthropic, OpenAI).
Configure Micrometer Tracing with an OTEL exporter.
Enable prompt/completion content observation via spring.ai.chat.observations.include-prompt=true and spring.ai.chat.observations.include-completion=true.
Send a chat request and inspect the exported spans.

Expected behavior
The spans exported to an OTEL backend should include:

gen_ai.input.messages — serialized list of prompt messages with role and content
gen_ai.output.messages — serialized list of completion messages with role and content parts
Tool call inputs/outputs as span attributes when tools are invoked

This aligns with the OpenTelemetry Generative AI semantic conventions.

Actual behavior
ChatModelPromptContentObservationHandler logs prompt/completion content to SLF4J only. No span attributes are set. OTEL backends receive traces with no LLM-specific content.

Minimal Complete Reproducible example
Users currently have to write a custom handler like this to fill the gap:

public class ChatModelSpanContentObservationHandler
implements ObservationHandler {

private final ObjectMapper objectMapper;

@Override
public void onStop(ChatModelObservationContext context) {
    TracingObservationHandler.TracingContext tracingContext =
            context.get(TracingObservationHandler.TracingContext.class);
    if (tracingContext == null) return;
    Span span = tracingContext.getSpan();
    if (span == null) return;
    tagPrompt(span, context);
    tagCompletion(span, context);
}

private void tagPrompt(Span span, ChatModelObservationContext context) {
    var instructions = context.getRequest().getInstructions();
    if (instructions == null || instructions.isEmpty()) return;
    try {
        List<Map<String, String>> messages = instructions.stream()
            .map(msg -> Map.of("role", msg.getMessageType().getValue(), 
                               "content", msg.getText()))
            .collect(Collectors.toList());
        span.tag("gen_ai.input.messages", objectMapper.writeValueAsString(messages));
    } catch (Exception e) {
        logger.info("Failed to serialize prompt for span tagging", e);
    }
}

// ... tagCompletion similarly

@Override
public boolean supportsContext(Observation.Context context) {
    return context instanceof ChatModelObservationContext;
}

}
This should not be necessary. Spring AI should either extend ChatModelPromptContentObservationHandler to also tag spans, or provide a dedicated ChatModelSpanContentObservationHandler that is auto-configured when tracing is on the classpath.

Suggested Fix
Extend ChatModelPromptContentObservationHandler#onStop to also call span.tag(...) when a TracingContext is present, similar to the workaround above.
Provide equivalent coverage for tool call observation contexts.
Auto-configure these handlers when micrometer-tracing is on the classpath, guarded by the existing spring.ai.chat.observations.include-prompt / include-completion flags.

[jewoodev]

Thanks for the detailed report, @AneshkaGoyal — I traced through the relevant code paths and wanted to share what I found, plus one question for the maintainers so the fix can be shaped in the direction they prefer.

What's missing vs. what's already there

It's worth splitting the gap into two cases, because the two paths differ today:

Chat content — genuinely missing on the span. ChatModelPromptContentObservationHandler.onStop() (spring-ai-model/.../chat/observation/ChatModelPromptContentObservationHandler.java:42-44) and its completion counterpart only log via SLF4J; TracingAwareLoggingObservationHandler reads TracingObservationHandler.TracingContext solely for log/trace correlation and never calls span.tag(...). AiObservationAttributes also doesn't define gen_ai.input.messages / gen_ai.output.messages yet, so even custom conventions can't reuse a shared constant.

Tool / function call content — this path already reaches the span, but under a non-standard key prefix. ToolCallingContentObservationFilter (spring-ai-model/.../tool/observation/ToolCallingContentObservationFilter.java:37-46) adds TOOL_CALL_ARGUMENTS / TOOL_CALL_RESULT as high-cardinality KVs which the Observation→Tracing bridge promotes to span attributes. The problem is purely the prefix: Spring AI exports spring.ai.tool.call.*, while the OTel GenAI semantic conventions expect gen_ai.tool.call.*, so OTel-native UIs (Datadog LLM Observability, Honeycomb GenAI views) don't recognize them out of the box.

For reference, the OTel GenAI spec (gen-ai spans) currently marks gen_ai.input.messages / gen_ai.output.messages / gen_ai.tool.call.{arguments,result} as Opt-In in Development stability, with the guidance "SHOULD NOT capture by default, but SHOULD provide an option for users to opt in." Spring AI's existing log-prompt / log-completion flags already match that opt-in philosophy on the logging side — only the wiring to the span path is missing.

One open question for the maintainers

Two implementation patterns seem reasonable, and I'd appreciate guidance on which fits the codebase's style:

(a) Dedicated ChatModelSpanContentObservationHandler — a new ObservationHandler<ChatModelObservationContext> that, on onStop(), pulls TracingContext, null-checks the span, and calls span.tag(...) with OTel-shaped JSON. Registered with @ConditionalOnClass(Tracer.class) + @ConditionalOnBean(Tracer.class) + an opt-in property.

(b) Dedicated ChatModelContentObservationFilter — mirroring the existing ToolCallingContentObservationFilter pattern: add gen_ai.input.messages / gen_ai.output.messages as high-cardinality KVs and let the tracing bridge promote them.

The trade-off I keep coming back to is cardinality / PII channel isolation: in (b) the high-cardinality KV ends up on the Observation.Context and is reachable by every handler attached to the same observation. Any user-defined handler or third-party integration that uses it as a metric tag would face cardinality blow-up on essentially unbounded prompt text, and any export-shaped handler could surface content into a channel users didn't intend. Pattern (a) keeps that content on the span only. This may also be why Spring AI keeps chat content in a ContentObservationHandler rather than in DefaultChatModelObservationConvention today.

Happy to follow whichever flag convention you prefer (reusing log-prompt / log-completion vs. a new include-content-style flag) in the PR.

Suggested scope split

To keep the first PR reviewable, scoping it to chat content only seems safest:

• This issue / first PR: add INPUT_MESSAGES / OUTPUT_MESSAGES to AiObservationAttributes, introduce the chosen pattern under ChatObservationAutoConfiguration with the opt-in property and Tracer guards.
• Separate follow-ups: aligning the tool side from spring.ai.tool.call.* to the OTel-standard gen_ai.tool.call.* prefix, and migrating gen_ai.system → gen_ai.provider.name (renamed in semconv v1.37.0, reference DO NOT MERGE - Module refactoring #2046).

PR #5756 touches the auto-config ordering / tracer conditions in this area; any new handler registration should probably land on top of that or coordinate with it.

I'm happy to send a PR within the next few days once the (a) vs (b) direction is clear — apologies if I've missed any prior discussion, please just redirect me if so.

[jewoodev]

Quick update: I left a review comment on PR #6057 with the concrete implementation concerns I found while comparing the PR with this issue: #6057 (comment)

The main points are:

• JSON serialization/escaping for gen_ai.input.messages
• micrometer-tracing optional classpath compatibility
• OTel GenAI content-capture opt-in behavior
• A scope question around shared AiObservationAttributes constants and the missing output-side counterpart

To avoid splitting the detailed implementation discussion, I think it would be best to continue the PR-specific review there.