Status: Draft for comment
Author: Anirudha (OpenSearch Observability maintainer)
Target repositories: opensearch-project/observability-stack (primary), opensearch-project/dashboards-observability, opensearch-project/sql, opensearch-project/data-prepper, opensearch-project/opentelemetry-demo
Site: otel.opensearch.org (Observability Stack documentation)
Related: Agentic AI evaluation platform RFC (dashboards-observability#2588), OpenTelemetry GenAI semantic conventions, OpenTelemetry code.*, vcs.*, deployment.*, and cicd.* semantic conventions
1. Summary
Observability was built for humans reading dashboards after something broke. The primary consumer is now an AI agent that writes, reviews, ships, and repairs code. That agent needs production runtime context before merge, not after an incident.
This RFC proposes that the OpenSearch Observability Stack become the open, OpenTelemetry-native runtime context layer for AI agents. It defines five open contracts and a partner conformance profile:
- Schema contract. Telemetry that is code-addressable: every span, log, metric, and deployment marker carries the OpenTelemetry attributes that let an agent map production behavior to a file, a function, a commit, and a pull request.
- Access contract. A versioned MCP tool set, PPL as the agent query language for logs and traces, PromQL for metrics, and a token-budgeted Context Pack response format.
- Skills contract. An open registry of Agent Skills (
SKILL.md) that teach any coding or SRE agent how to use the stack correctly.
- Workflow contract. Reference shift-left recipes (blast radius, pre-merge comparison, post-deploy verification, issue-to-PR remediation, health reports, cold-code cleanup, AI token-cost regression) runnable on GitHub Actions, Claude Code, Cursor, Kiro, or any MCP client.
- Loop contract. Every agent action is itself traced with GenAI semantic conventions and linked to evaluation results, so the stack observes, evaluates, and improves the agents that use it.
Partners that produce runtime data (runtime code sensors, error monitors, APM agents, CI/CD systems, coding agents) integrate through these contracts rather than through bespoke plugins. A "Shift-Left Ready" conformance profile makes the integration testable and listable on otel.opensearch.org.
2. Motivation
2.1 The consumer of observability changed
AI coding agents now write a large share of production code. They plan, edit, test, and review, and their assistance stops at merge. What they lack is what production knows: which functions carry traffic, which paths are hot, which exceptions are new, which endpoint regressed after the last deploy. Static analysis cannot answer these questions. Only runtime data can.
2.2 The market has converged on one pattern
Across function-level runtime sensors, error monitoring platforms, and application performance vendors, the same architecture has emerged in the last twelve months:
| Capability |
Pattern |
| Data |
Function-level and endpoint-level runtime data from production, often unsampled, mapped to source file and line |
| Agent access |
A first-party MCP server (local and hosted) plus a query API an agent can call directly |
| Pre-merge |
Pull request risk checks that compare the changed functions against live production behavior |
| Post-deploy |
Release verification and rollback recommendations from production deltas |
| Remediation |
A pipeline from detected issue to root cause to an opened pull request |
| Runners |
Recipes packaged for GitHub Actions, IDE automations, and scheduled coding-agent tasks |
| Self-observation |
Monitoring of the agents and MCP servers themselves, including LLM calls, tokens, and tool executions |
Each implementation is proprietary: proprietary schema, proprietary query language, proprietary MCP tools. A team that adopts one cannot move its agent workflows to another, and a team that runs OpenSearch for observability cannot use its own data in any of them.
2.3 OpenSearch's position
The Observability Stack already has the foundation: OTLP ingestion, OpenTelemetry semantic conventions as the canonical schema (including GenAI conventions for agent traces), PPL with pattern mining and anomaly detection built into the query pipeline, Prometheus-compatible metrics, and a Claude Code plugin with an MCP server and skills scaffold in claude-code-observability-plugin/. What is missing is the set of contracts that turn this into something an agent, and a partner, can build on without asking a maintainer.
The opportunity is to be the open substrate for shift-left observability: the place where runtime context from any source lands in a standard shape and is served to any agent through standard interfaces.
3. Goals and non-goals
Goals
- An AI coding agent working in any repository can obtain production runtime context for the code it is changing in one tool call, with a bounded token cost.
- A pull request can be gated on production evidence (traffic, latency, errors, token cost) using only OpenSearch and open-source components.
- A production issue can flow to a remediation pull request, and the fix can be verified against production, with every agent step traced and evaluable.
- A third party can integrate a runtime data source or an agent workflow by conforming to published contracts and passing a conformance suite.
Non-goals
- Building a first-party function-level runtime sensor. Function-level data enters through OTLP from partners and from OpenTelemetry instrumentation.
- Replacing OpenSearch Dashboards or the Investigations experience. Those remain the human surfaces; this RFC defines the agent surfaces.
- Prescribing a single coding agent or CI system.
4. Proposal overview
Coding / SRE agents (Claude Code, Cursor, Kiro, Codex, custom)
| MCP tools + PPL + PromQL + Skills
v
+-----------------------------------------------------------+
| Shift-Left Access Layer (observability-stack MCP server) |
| get_runtime_context | blast_radius | compare_windows |
| search_issues | get_deployments | query_ppl | query_promql|
| list_skills | record_agent_action |
+-----------------------------------------------------------+
|
v
+-----------------------------------------------------------+
| OpenSearch + Prometheus (OTel semconv schema profile) |
| spans (code.*, vcs.*, deployment.*, gen_ai.*) |
| logs (exception.*, trace context) |
| metrics (code.function.* from sensors, RED, gen_ai.*) |
| deployment markers | agent-actions | evaluation results |
+-----------------------------------------------------------+
^
| OTLP (schema profile) | federated source | recipes
Runtime data producers: OTel SDKs, runtime code sensors, error
monitors, APM agents, CI/CD systems, coding agents (self-trace)
5. Detailed design
5.1 Schema contract: code-addressable telemetry
The stack already indexes OpenTelemetry attributes as-is. This section defines the Shift-Left Schema Profile, the minimum attribute set that makes telemetry addressable from a repository. All names are OpenTelemetry semantic conventions unless marked proposed.
| Attribute |
Signal |
Purpose |
Status |
service.name, service.version, service.instance.id |
all |
Service identity and release |
Stable |
deployment.environment.name |
all |
Separate production from preview and staging cohorts |
Stable |
code.function.name, code.file.path, code.line.number, code.namespace |
spans, logs, metrics |
Map behavior to source |
Stable |
code.stacktrace, exception.type, exception.message, exception.stacktrace |
span events, logs |
Error forensics |
Stable |
vcs.repository.url.full, vcs.ref.head.revision, vcs.ref.head.name |
resource |
Tie telemetry to a commit and branch |
Development |
vcs.change.id, vcs.change.state |
resource |
Tag preview or canary telemetry with the pull request under test |
Development |
deployment.id, deployment.name, deployment.status |
deployment markers |
Before/after comparison windows |
Development |
cicd.pipeline.name, cicd.pipeline.run.id |
deployment markers, agent actions |
Link gates and runs to CI |
Development |
gen_ai.operation.name, gen_ai.agent.name, gen_ai.tool.name, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.request.model |
spans |
Observe AI applications and the shift-left agents themselves |
Stable / Development |
gen_ai.evaluation.result, test.case.id, test.suite.run.id |
spans, evaluation records |
Link agent output to evaluation outcomes |
Proposed in OTel GenAI SIG |
code.function.caller.name |
metrics, spans |
Call-graph edge for function-level metrics |
Proposed; upstream via OTel semconv SIG |
Function-level metrics. Runtime sensors that capture unsampled function-level data export it as OpenTelemetry metrics rather than as a proprietary table. Proposed instrument set (experimental namespace, to be confirmed with the OTel semantic conventions SIG):
| Instrument |
Type |
Unit |
Attributes |
code.function.calls |
Counter |
{call} |
code.function.name, code.file.path, code.namespace, code.function.caller.name, service.name, deployment.environment.name |
code.function.duration |
Histogram |
s |
same |
code.function.errors |
Counter |
{error} |
same plus error.type |
These land in Prometheus through the Collector's OTLP HTTP path with promote_resource_attributes already configured in the stack, and in OpenSearch for long-term correlation.
Deployment markers. A deployment is recorded as an OTel log record (event name deployment) carrying deployment.*, vcs.*, service.*, and cicd.* attributes. Data Prepper routes these to deployments-otel-v1-*. A GitHub Action and a curl one-liner are provided as reference emitters.
Pipeline changes. The Collector transform already flattens code.function.name to code_function_name. The profile extends this to the full code.*, vcs.*, and deployment.* set. Data Prepper index templates add keyword mappings for attributes.code.function.name, attributes.code.file.path, resource.vcs.change.id, resource.vcs.ref.head.revision, and resource.deployment.id. Field-level security policies cover attributes.gen_ai.*.arguments, body, and exception.message for PII control.
5.2 Access contract: MCP tool set, PPL, PromQL, Context Pack
The MCP server in claude-code-observability-plugin/ becomes the reference implementation of the Shift-Left Tool Contract v0. The contract is small, read-mostly, and stable.
| Tool |
Purpose |
Inputs |
Output |
get_runtime_context |
Production behavior for the code an agent is touching |
repo, ref, one of paths[] / functions[] / services[], environment, window, budget_tokens |
Context Pack |
blast_radius |
Rank changed functions and endpoints by production exposure, with callers and downstream services |
diff or changed_files[], environment, window |
Ranked list with traffic, p90, error rate, callers, service-map neighbors |
compare_windows |
Baseline vs candidate deltas for a selector |
selector (functions, services, endpoints, or gen_ai.agent.name), baseline (window or cohort), candidate (window, deployment.id, or vcs.change.id) |
Deltas for rate, errors, p50/p90/p99, tokens, cost, with sample sizes |
search_issues |
New and regressed error fingerprints |
services[], since, environment |
Clustered patterns from PPL patterns with counts, first seen, last seen, example trace |
get_deployments |
Deployment markers |
service, since |
Marker list with deployment.id, revision, PR, status |
get_trace, get_logs |
Forensics for a specific trace |
trace_id or PPL filter |
Span tree and correlated logs |
query_ppl |
Escape hatch for logs and traces |
query, max_rows, max_bytes |
Rows plus the executed query for provenance |
query_promql |
Escape hatch for metrics |
query, start, end, step |
Series |
list_skills, get_skill |
Discover and load skills from the registry |
name |
SKILL.md content |
record_agent_action |
Closed loop: write an agent action as an OTel event |
action, target (fingerprint, function, PR), gen_ai.* context |
Event id |
Transport and auth. Local stdio for IDE and terminal agents; streamable HTTP for hosted use. Authentication uses the OpenSearch security plugin: a shift_left_reader role with read access to otel-v1-apm-span-*, otel-v1-apm-log-*, deployments-otel-v1-*, and metrics indices, and a shift_left_agent_writer role with write access only to agent-actions-otel-v1-*. Managed deployments use SigV4 or OIDC.
Response budgets. Every tool accepts max_rows (default 200) and max_bytes (default 256 KB). get_runtime_context accepts budget_tokens and truncates by priority: errors, then regressions, then traffic, then trend.
PPL as the agent query language. PPL is pipe-based, readable, and includes patterns, ml, join, and span in the same pipeline, which is why it is the default for logs and traces. Reference queries used by the tools:
Blast radius for functions touched by a diff:
source = otel-v1-apm-span-*
| where `resource.deployment.environment.name` = 'production'
AND `attributes.code.function.name` IN ('checkout.applyDiscount', 'checkout.computeTax')
| eval duration_ms = durationInNanos / 1000000
| stats count() as calls,
percentile(duration_ms, 90) as p90_ms,
sum(if(`status.code` = 2, 1, 0)) as errors
by `attributes.code.function.name`, serviceName
| eval error_rate = errors / calls
| sort - calls
Upstream callers of a changed function:
source = otel-v1-apm-span-*
| where `attributes.code.function.name` = 'checkout.applyDiscount'
| join left=c right=p ON c.parentSpanId = p.spanId otel-v1-apm-span-*
| stats count() as calls by p.serviceName, p.name
| sort - calls
| head 20
Candidate (pull request preview) vs production baseline:
source = otel-v1-apm-span-*
| where `resource.vcs.change.id` = '4812'
OR `resource.deployment.environment.name` = 'production'
| eval cohort = if(`resource.vcs.change.id` = '4812', 'candidate', 'baseline'),
duration_ms = durationInNanos / 1000000
| stats count() as calls,
percentile(duration_ms, 90) as p90_ms,
sum(if(`status.code` = 2, 1, 0)) as errors
by cohort, `attributes.code.function.name`
New error fingerprints since a deployment:
source = otel-v1-apm-log-*
| where severityNumber >= 17 AND `@timestamp` > '2026-09-01 14:00:00'
| patterns body method=brain mode=aggregation by serviceName
| sort - pattern_count
| head 20
Token-cost regression for an AI application change:
source = otel-v1-apm-span-*
| where `attributes.gen_ai.operation.name` IN ('chat', 'invoke_agent')
| eval cohort = if(`resource.vcs.change.id` = '4812', 'candidate', 'baseline'),
tokens = `attributes.gen_ai.usage.input_tokens` + `attributes.gen_ai.usage.output_tokens`
| stats avg(tokens) as avg_tokens,
percentile(durationInNanos, 90) as p90_ns,
count() as calls
by cohort, `attributes.gen_ai.agent.name`
Function-level metrics from a conformant sensor, in PromQL:
sum by (code_function_name) (rate(code_function_calls_total{service_name="checkout"}[5m]))
histogram_quantile(0.9,
sum by (le, code_function_name) (rate(code_function_duration_seconds_bucket{service_name="checkout"}[5m])))
Context Pack format (v0). A compact JSON document designed to be pasted into a model context:
{
"schema": "opensearch.shift-left.context-pack/v0",
"scope": {"repo": "acme/checkout", "ref": "feat/discounts", "environment": "production", "window": "7d"},
"budget": {"tokens": 4000, "truncated": false},
"functions": [{
"code.function.name": "checkout.applyDiscount",
"code.file.path": "src/checkout/discount.ts",
"calls_per_min": 412.3, "p50_ms": 12, "p90_ms": 88, "p99_ms": 410, "error_rate": 0.012,
"top_exceptions": [{"fingerprint": "f3a1", "exception.type": "TypeError", "count": 91, "first_seen": "2026-08-29T02:14Z"}],
"callers": ["api.postCheckout"], "callees": ["pricing.lookupRule", "tax.compute"],
"last_deployment": {"deployment.id": "d-2291", "vcs.ref.head.revision": "9c1e2f", "at": "2026-08-31T19:02Z"},
"trend_vs_prior_window": {"calls": "+8%", "p90_ms": "+35%", "error_rate": "flat"}
}],
"services": [{"service.name": "checkout", "rate_rps": 61.2, "error_rate": 0.004, "p90_ms": 140, "slo_burn_rate_1h": 0.6}],
"evidence": [{"tool": "query_ppl", "query": "source = otel-v1-apm-span-* | where ..."}],
"links": [{"label": "Traces", "url": "https://dashboards.example/app/observability-traces#/..."}]
}
The evidence array carries the exact queries used, so an agent (or a reviewer) can re-run and verify rather than trust a summary.
Agent-readable documentation. otel.opensearch.org publishes llms.txt at the root, a Markdown rendering of every page, and an OpenAPI description of the PPL, PromQL, and MCP endpoints.
5.3 Skills contract: the open skills registry
Agent Skills (SKILL.md with frontmatter name, description, allowed-tools) are the portable way to teach an agent how to use the stack. The registry lives in observability-stack/skills/ with a generated index.json served from otel.opensearch.org so any MCP client can discover skills through list_skills.
| Skill |
Status |
Content |
ppl-reference |
Exists in plugin |
Full PPL command and function reference with curl examples |
logs |
Exists in plugin |
Severity filtering, trace correlation, error patterns, volume analysis |
apm-red |
Exists in plugin |
Rate, errors, duration in PromQL and PPL |
correlation |
Exists in plugin |
Cross-signal correlation on OTel fields and exemplars |
slo-sli |
Exists in plugin |
Recording rules, error budgets, burn-rate alerts |
shift-left/blast-radius |
New |
How to compute and present pull request exposure |
shift-left/pre-merge-comparison |
New |
Cohort tagging with vcs.change.id, thresholds, verdict format |
shift-left/post-deploy-verify |
New |
Deployment markers, comparison windows, rollback recommendation |
shift-left/remediation |
New |
Issue to root cause to PR, with required evidence and self-tracing |
shift-left/ai-cost-regression |
New |
Token, latency, and evaluation deltas for AI application changes |
agent-observability |
New |
Instrumenting the shift-left agent itself with GenAI conventions |
Partners contribute skills under skills/partners/<name>/ after passing conformance. Skills are Apache 2.0.
5.4 Workflow contract: reference recipes and runners
Each recipe is a documented, testable flow with declared inputs, tools, verdict format, and thresholds.
| Recipe |
Trigger |
Tools |
Output |
Default runner |
| PR blast radius |
pull_request opened or updated |
blast_radius, get_runtime_context |
PR comment with ranked exposure table and a risk label (low, medium, high) |
GitHub Action |
| Pre-merge behavioral comparison |
Preview environment deployed with vcs.change.id |
compare_windows |
Pass/fail check on error rate, p90, and token deltas against configured thresholds |
GitHub Action |
| Post-deploy verification |
Deployment marker received |
get_deployments, compare_windows, search_issues |
Verdict at 15 and 60 minutes; rollback recommendation with evidence |
Scheduled agent task or Alertmanager webhook |
| Issue to remediation PR |
New error fingerprint in production |
search_issues, get_trace, get_runtime_context, record_agent_action |
Draft PR with root cause, evidence links, and test |
Claude Code, Cursor, or Kiro agent |
| Weekly service health |
Schedule |
get_runtime_context, query_promql |
Report of regressions, SLO burn, top new exceptions, owned by service |
Claude Code scheduled task |
| Cold-code cleanup |
Schedule |
query_ppl (zero calls in 90 days) |
PR proposing removal, with production evidence |
Scheduled agent task |
| AI token-cost regression |
Prompt, model, or tool change in an AI application |
compare_windows on gen_ai.* |
Pass/fail on tokens per operation, latency, and evaluation score |
GitHub Action |
Reference implementations.
opensearch-project/shift-left-action: a composite GitHub Action wrapping the MCP server in headless mode, with blast-radius, compare, and verify subcommands and a PR comment template.
- Claude Code plugin commands:
/blast-radius, /compare-pr, /verify-deploy, /remediate <fingerprint>, /health-report.
- Cursor and Kiro: equivalent rules and hooks in
runners/.
- A
deployment-marker script and GitHub Action step that emit the marker event on every deploy.
5.5 Loop contract: observe, evaluate, improve the agents
Shift-left agents are AI agents. They are instrumented with the same GenAI conventions the stack already serves:
- Every recipe run is a trace:
gen_ai.operation.name = invoke_agent, gen_ai.agent.name = <recipe>, tool calls as execute_tool spans with gen_ai.tool.name, token usage on every model call.
record_agent_action writes the business outcome (PR opened, gate passed, rollback recommended) with vcs.change.id, target code.function.name, and the originating fingerprint.
- When post-deploy verification passes or fails for a remediation PR, an evaluation record is written with
gen_ai.evaluation.result linked to the remediation trace.
This gives the Investigations experience the data to answer "which agent fixed which fingerprint, at what token cost, and did it hold", and gives the evaluation platform (RFC 2588) online signals for remediation quality. The MCP server monitors itself the same way.
5.6 Partner integration model and conformance
Three integration surfaces, each defined by the contracts above:
| Surface |
Who |
What they implement |
| Ingest |
Runtime code sensors, error monitors, APM agents, CI/CD systems |
Export OTLP conforming to the Schema Profile: code.* on spans, logs, and function metrics; vcs.* and deployment.* on resources; deployment markers; PII scrubbed at source |
| Context |
Vendors that keep data in place |
Expose an MCP server implementing get_runtime_context, blast_radius, and compare_windows with the Context Pack format, or register a federated data source queryable from PPL. Data stays in place; only context moves |
| Workflow |
Coding agents, IDEs, CI platforms, automation vendors |
Ship recipes and runners; emit agent traces with GenAI conventions; call record_agent_action |
"Shift-Left Ready" conformance profile. A test suite in observability-stack/conformance/ verifies:
- Required attributes present and correctly typed on a sample workload.
- Deployment markers emitted and indexed.
- Tool contract responses validate against the JSON schemas, including budget enforcement and
evidence.
- Function-level metrics resolve in PromQL with expected labels.
- Agent self-traces present for at least one recipe.
- PII policy: no raw request payloads in
attributes.*.arguments without a documented scrubbing rule.
Conformant integrations are listed on otel.opensearch.org with their surface level. The listing is a project artifact, not a commercial program; commercial partnership terms are outside this RFC.
5.7 Security and governance
- Read-only by default. Only
record_agent_action writes, and only to agent-actions-otel-v1-*.
- Least privilege via OpenSearch security roles and field-level security. Index patterns and redaction rules are documented per role.
- Every MCP call is audited twice: in the OpenSearch audit log and as a span from the MCP server itself.
- Response budgets and query timeouts are enforced server-side.
query_ppl rejects delete-class commands and enforces a wall-clock limit.
- Data residency: federated context sources let partners and customers keep raw data in place, which enterprise VoC identifies as a hard requirement at petabyte scale.
5.8 Documentation and site
otel.opensearch.org adds a top-level Shift Left section:
- Overview and the five contracts
- Get started in five minutes: Claude Code, Cursor, Kiro, GitHub Actions
- Schema Profile reference
- MCP Tool Contract reference (with JSON schemas)
- Skills registry
- Workflow recipes
- Partner integration and conformance
llms.txt and per-page Markdown for agents
6. Reference journeys
A developer changes a hot function. In Claude Code, the agent calls get_runtime_context with the changed paths. The Context Pack shows checkout.applyDiscount at 412 calls per minute, p90 up 35 percent since the last deploy, and a TypeError fingerprint first seen two days ago. The agent adds a guard, writes a test that reproduces the fingerprint's arguments shape, and opens a PR citing the evidence queries.
A pull request is gated. The GitHub Action runs blast_radius on the diff and labels the PR high because it touches two functions on the checkout path. A preview deploy tagged vcs.change.id=4812 runs canary traffic; compare_windows reports p90 within threshold and no new fingerprints. The check passes.
A production issue becomes a PR and is verified. A new fingerprint appears. The remediation recipe pulls the trace, correlated logs, and runtime context, opens a draft PR with root cause and evidence, and records the action. After deploy, post-deploy verification confirms the fingerprint stopped and writes an evaluation result linked to the remediation trace.
7. Alternatives considered
| Alternative |
Decision |
| Proprietary REST API for agents instead of MCP |
Rejected. MCP is the transport coding agents already speak; REST remains available underneath |
| SQL as the primary agent query language |
PPL is primary for logs and traces because patterns, ml, join, and span compose in one pipeline. SQL is supported for partners that expose SQL sources |
| Build a first-party function-level sensor |
Rejected for this RFC. Function-level data arrives via OTLP from partners and OpenTelemetry instrumentation; the project owns the schema and the access layer |
| One plugin per coding agent |
Replaced by MCP plus skills. Runners are thin |
| Dashboard-first agentic UX only |
Complementary. Investigations serves humans; this RFC serves agents. Both read the same data |
| Vendor-specific attribute names |
Rejected. Everything upstreams to OpenTelemetry semantic conventions; proposed attributes are marked and tracked |
8. Rollout
| Phase |
Window |
Deliverables |
| 0 |
Sep to Oct 2026 |
RFC accepted; Schema Profile doc; Collector and Data Prepper template changes; Tool Contract v0 in the plugin MCP server; new shift-left skills; llms.txt on the site |
| 1 |
Nov to Dec 2026 |
shift-left-action with blast radius and PR comment; deployment marker emitter; compare_windows; first external integration passes conformance levels 1 and 2 |
| 2 |
Q1 2027 |
Post-deploy verification with rollback recommendation; remediation recipe with self-tracing and evaluation linkage; AI token-cost regression gate; Investigations reads agent actions |
| 3 |
Q2 2027 |
Conformance program and listings on otel.opensearch.org; federated context sources; upstream proposals for code.function.* metrics and code.function.caller.name submitted to OTel |
9. Success metrics
- Weekly active MCP clients against the stack, and Context Packs served per week.
- Pull requests annotated or gated per week; false-positive rate of
high labels as judged by reviewers.
- Median time from fingerprint first seen to remediation PR opened; percentage of remediation PRs merged; percentage that hold at 7 days.
- Tokens consumed per successful root cause, tracked toward the existing 10x reduction target.
- Number of conformant integrations by surface level; skills installed from the registry.
10. Open questions
- Cardinality and cost of unsampled function-level metrics: aggregate in the Collector, in Data Prepper, or accept full cardinality in Prometheus with recording rules.
- Call-graph representation: derive from span parentage only, or maintain an
edges index for sensors that report caller relationships without spans.
- Hosted MCP authentication for managed deployments: SigV4 passthrough vs OIDC device flow.
- Whether
record_agent_action should be an OTel log event only, or also a span link to the originating fingerprint.
- Threshold defaults for gates, and whether they should derive from SLOs already defined in the stack.
- Governance of partner-contributed skills and recipes: review process, versioning, and deprecation.
- Naming and placement of the proposed
code.function.* instruments in upstream OpenTelemetry.
11. Request for comment
Feedback is requested on the Tool Contract v0 surface, the Schema Profile attribute list, the Context Pack format, and the conformance checklist. Prototype PRs for Phase 0 will follow acceptance.
Status: Draft for comment
Author: Anirudha (OpenSearch Observability maintainer)
Target repositories:
opensearch-project/observability-stack(primary),opensearch-project/dashboards-observability,opensearch-project/sql,opensearch-project/data-prepper,opensearch-project/opentelemetry-demoSite: otel.opensearch.org (Observability Stack documentation)
Related: Agentic AI evaluation platform RFC (
dashboards-observability#2588), OpenTelemetry GenAI semantic conventions, OpenTelemetrycode.*,vcs.*,deployment.*, andcicd.*semantic conventions1. Summary
Observability was built for humans reading dashboards after something broke. The primary consumer is now an AI agent that writes, reviews, ships, and repairs code. That agent needs production runtime context before merge, not after an incident.
This RFC proposes that the OpenSearch Observability Stack become the open, OpenTelemetry-native runtime context layer for AI agents. It defines five open contracts and a partner conformance profile:
SKILL.md) that teach any coding or SRE agent how to use the stack correctly.Partners that produce runtime data (runtime code sensors, error monitors, APM agents, CI/CD systems, coding agents) integrate through these contracts rather than through bespoke plugins. A "Shift-Left Ready" conformance profile makes the integration testable and listable on otel.opensearch.org.
2. Motivation
2.1 The consumer of observability changed
AI coding agents now write a large share of production code. They plan, edit, test, and review, and their assistance stops at merge. What they lack is what production knows: which functions carry traffic, which paths are hot, which exceptions are new, which endpoint regressed after the last deploy. Static analysis cannot answer these questions. Only runtime data can.
2.2 The market has converged on one pattern
Across function-level runtime sensors, error monitoring platforms, and application performance vendors, the same architecture has emerged in the last twelve months:
Each implementation is proprietary: proprietary schema, proprietary query language, proprietary MCP tools. A team that adopts one cannot move its agent workflows to another, and a team that runs OpenSearch for observability cannot use its own data in any of them.
2.3 OpenSearch's position
The Observability Stack already has the foundation: OTLP ingestion, OpenTelemetry semantic conventions as the canonical schema (including GenAI conventions for agent traces), PPL with pattern mining and anomaly detection built into the query pipeline, Prometheus-compatible metrics, and a Claude Code plugin with an MCP server and skills scaffold in
claude-code-observability-plugin/. What is missing is the set of contracts that turn this into something an agent, and a partner, can build on without asking a maintainer.The opportunity is to be the open substrate for shift-left observability: the place where runtime context from any source lands in a standard shape and is served to any agent through standard interfaces.
3. Goals and non-goals
Goals
Non-goals
4. Proposal overview
5. Detailed design
5.1 Schema contract: code-addressable telemetry
The stack already indexes OpenTelemetry attributes as-is. This section defines the Shift-Left Schema Profile, the minimum attribute set that makes telemetry addressable from a repository. All names are OpenTelemetry semantic conventions unless marked proposed.
service.name,service.version,service.instance.iddeployment.environment.namecode.function.name,code.file.path,code.line.number,code.namespacecode.stacktrace,exception.type,exception.message,exception.stacktracevcs.repository.url.full,vcs.ref.head.revision,vcs.ref.head.namevcs.change.id,vcs.change.statedeployment.id,deployment.name,deployment.statuscicd.pipeline.name,cicd.pipeline.run.idgen_ai.operation.name,gen_ai.agent.name,gen_ai.tool.name,gen_ai.usage.input_tokens,gen_ai.usage.output_tokens,gen_ai.request.modelgen_ai.evaluation.result,test.case.id,test.suite.run.idcode.function.caller.nameFunction-level metrics. Runtime sensors that capture unsampled function-level data export it as OpenTelemetry metrics rather than as a proprietary table. Proposed instrument set (experimental namespace, to be confirmed with the OTel semantic conventions SIG):
code.function.calls{call}code.function.name,code.file.path,code.namespace,code.function.caller.name,service.name,deployment.environment.namecode.function.durationscode.function.errors{error}error.typeThese land in Prometheus through the Collector's OTLP HTTP path with
promote_resource_attributesalready configured in the stack, and in OpenSearch for long-term correlation.Deployment markers. A deployment is recorded as an OTel log record (event name
deployment) carryingdeployment.*,vcs.*,service.*, andcicd.*attributes. Data Prepper routes these todeployments-otel-v1-*. A GitHub Action and acurlone-liner are provided as reference emitters.Pipeline changes. The Collector transform already flattens
code.function.nametocode_function_name. The profile extends this to the fullcode.*,vcs.*, anddeployment.*set. Data Prepper index templates add keyword mappings forattributes.code.function.name,attributes.code.file.path,resource.vcs.change.id,resource.vcs.ref.head.revision, andresource.deployment.id. Field-level security policies coverattributes.gen_ai.*.arguments,body, andexception.messagefor PII control.5.2 Access contract: MCP tool set, PPL, PromQL, Context Pack
The MCP server in
claude-code-observability-plugin/becomes the reference implementation of the Shift-Left Tool Contract v0. The contract is small, read-mostly, and stable.get_runtime_contextrepo,ref, one ofpaths[]/functions[]/services[],environment,window,budget_tokensblast_radiusdifforchanged_files[],environment,windowcompare_windowsselector(functions, services, endpoints, orgen_ai.agent.name),baseline(window or cohort),candidate(window,deployment.id, orvcs.change.id)search_issuesservices[],since,environmentpatternswith counts, first seen, last seen, example traceget_deploymentsservice,sincedeployment.id, revision, PR, statusget_trace,get_logstrace_idor PPL filterquery_pplquery,max_rows,max_bytesquery_promqlquery,start,end,steplist_skills,get_skillnameSKILL.mdcontentrecord_agent_actionaction,target(fingerprint, function, PR),gen_ai.*contextTransport and auth. Local stdio for IDE and terminal agents; streamable HTTP for hosted use. Authentication uses the OpenSearch security plugin: a
shift_left_readerrole with read access tootel-v1-apm-span-*,otel-v1-apm-log-*,deployments-otel-v1-*, and metrics indices, and ashift_left_agent_writerrole with write access only toagent-actions-otel-v1-*. Managed deployments use SigV4 or OIDC.Response budgets. Every tool accepts
max_rows(default 200) andmax_bytes(default 256 KB).get_runtime_contextacceptsbudget_tokensand truncates by priority: errors, then regressions, then traffic, then trend.PPL as the agent query language. PPL is pipe-based, readable, and includes
patterns,ml,join, andspanin the same pipeline, which is why it is the default for logs and traces. Reference queries used by the tools:Blast radius for functions touched by a diff:
Upstream callers of a changed function:
Candidate (pull request preview) vs production baseline:
New error fingerprints since a deployment:
Token-cost regression for an AI application change:
Function-level metrics from a conformant sensor, in PromQL:
Context Pack format (v0). A compact JSON document designed to be pasted into a model context:
{ "schema": "opensearch.shift-left.context-pack/v0", "scope": {"repo": "acme/checkout", "ref": "feat/discounts", "environment": "production", "window": "7d"}, "budget": {"tokens": 4000, "truncated": false}, "functions": [{ "code.function.name": "checkout.applyDiscount", "code.file.path": "src/checkout/discount.ts", "calls_per_min": 412.3, "p50_ms": 12, "p90_ms": 88, "p99_ms": 410, "error_rate": 0.012, "top_exceptions": [{"fingerprint": "f3a1", "exception.type": "TypeError", "count": 91, "first_seen": "2026-08-29T02:14Z"}], "callers": ["api.postCheckout"], "callees": ["pricing.lookupRule", "tax.compute"], "last_deployment": {"deployment.id": "d-2291", "vcs.ref.head.revision": "9c1e2f", "at": "2026-08-31T19:02Z"}, "trend_vs_prior_window": {"calls": "+8%", "p90_ms": "+35%", "error_rate": "flat"} }], "services": [{"service.name": "checkout", "rate_rps": 61.2, "error_rate": 0.004, "p90_ms": 140, "slo_burn_rate_1h": 0.6}], "evidence": [{"tool": "query_ppl", "query": "source = otel-v1-apm-span-* | where ..."}], "links": [{"label": "Traces", "url": "https://dashboards.example/app/observability-traces#/..."}] }The
evidencearray carries the exact queries used, so an agent (or a reviewer) can re-run and verify rather than trust a summary.Agent-readable documentation. otel.opensearch.org publishes
llms.txtat the root, a Markdown rendering of every page, and an OpenAPI description of the PPL, PromQL, and MCP endpoints.5.3 Skills contract: the open skills registry
Agent Skills (
SKILL.mdwith frontmattername,description,allowed-tools) are the portable way to teach an agent how to use the stack. The registry lives inobservability-stack/skills/with a generatedindex.jsonserved from otel.opensearch.org so any MCP client can discover skills throughlist_skills.ppl-referencecurlexampleslogsapm-redcorrelationslo-slishift-left/blast-radiusshift-left/pre-merge-comparisonvcs.change.id, thresholds, verdict formatshift-left/post-deploy-verifyshift-left/remediationshift-left/ai-cost-regressionagent-observabilityPartners contribute skills under
skills/partners/<name>/after passing conformance. Skills are Apache 2.0.5.4 Workflow contract: reference recipes and runners
Each recipe is a documented, testable flow with declared inputs, tools, verdict format, and thresholds.
pull_requestopened or updatedblast_radius,get_runtime_contextlow,medium,high)vcs.change.idcompare_windowsget_deployments,compare_windows,search_issuessearch_issues,get_trace,get_runtime_context,record_agent_actionget_runtime_context,query_promqlquery_ppl(zero calls in 90 days)compare_windowsongen_ai.*Reference implementations.
opensearch-project/shift-left-action: a composite GitHub Action wrapping the MCP server in headless mode, withblast-radius,compare, andverifysubcommands and a PR comment template./blast-radius,/compare-pr,/verify-deploy,/remediate <fingerprint>,/health-report.runners/.deployment-markerscript and GitHub Action step that emit the marker event on every deploy.5.5 Loop contract: observe, evaluate, improve the agents
Shift-left agents are AI agents. They are instrumented with the same GenAI conventions the stack already serves:
gen_ai.operation.name = invoke_agent,gen_ai.agent.name = <recipe>, tool calls asexecute_toolspans withgen_ai.tool.name, token usage on every model call.record_agent_actionwrites the business outcome (PR opened, gate passed, rollback recommended) withvcs.change.id, targetcode.function.name, and the originating fingerprint.gen_ai.evaluation.resultlinked to the remediation trace.This gives the Investigations experience the data to answer "which agent fixed which fingerprint, at what token cost, and did it hold", and gives the evaluation platform (RFC 2588) online signals for remediation quality. The MCP server monitors itself the same way.
5.6 Partner integration model and conformance
Three integration surfaces, each defined by the contracts above:
code.*on spans, logs, and function metrics;vcs.*anddeployment.*on resources; deployment markers; PII scrubbed at sourceget_runtime_context,blast_radius, andcompare_windowswith the Context Pack format, or register a federated data source queryable from PPL. Data stays in place; only context movesrecord_agent_action"Shift-Left Ready" conformance profile. A test suite in
observability-stack/conformance/verifies:evidence.attributes.*.argumentswithout a documented scrubbing rule.Conformant integrations are listed on otel.opensearch.org with their surface level. The listing is a project artifact, not a commercial program; commercial partnership terms are outside this RFC.
5.7 Security and governance
record_agent_actionwrites, and only toagent-actions-otel-v1-*.query_pplrejectsdelete-class commands and enforces a wall-clock limit.5.8 Documentation and site
otel.opensearch.org adds a top-level Shift Left section:
llms.txtand per-page Markdown for agents6. Reference journeys
A developer changes a hot function. In Claude Code, the agent calls
get_runtime_contextwith the changed paths. The Context Pack showscheckout.applyDiscountat 412 calls per minute, p90 up 35 percent since the last deploy, and aTypeErrorfingerprint first seen two days ago. The agent adds a guard, writes a test that reproduces the fingerprint's arguments shape, and opens a PR citing the evidence queries.A pull request is gated. The GitHub Action runs
blast_radiuson the diff and labels the PRhighbecause it touches two functions on the checkout path. A preview deploy taggedvcs.change.id=4812runs canary traffic;compare_windowsreports p90 within threshold and no new fingerprints. The check passes.A production issue becomes a PR and is verified. A new fingerprint appears. The remediation recipe pulls the trace, correlated logs, and runtime context, opens a draft PR with root cause and evidence, and records the action. After deploy, post-deploy verification confirms the fingerprint stopped and writes an evaluation result linked to the remediation trace.
7. Alternatives considered
patterns,ml,join, andspancompose in one pipeline. SQL is supported for partners that expose SQL sources8. Rollout
llms.txton the siteshift-left-actionwith blast radius and PR comment; deployment marker emitter;compare_windows; first external integration passes conformance levels 1 and 2code.function.*metrics andcode.function.caller.namesubmitted to OTel9. Success metrics
highlabels as judged by reviewers.10. Open questions
edgesindex for sensors that report caller relationships without spans.record_agent_actionshould be an OTel log event only, or also a span link to the originating fingerprint.code.function.*instruments in upstream OpenTelemetry.11. Request for comment
Feedback is requested on the Tool Contract v0 surface, the Schema Profile attribute list, the Context Pack format, and the conformance checklist. Prototype PRs for Phase 0 will follow acceptance.