durable-harness: Agent Runs tracing on @vercel/agent-runs

[chadhietala]

agentRuns() from durable-harness/instrumentation/otel now records sessions with @vercel/agent-runs (vercel/eve#4910), the library eve uses for Agent Runs, instead of durable-harness's own span code. The library writes the turn, step, model and tool spans in the schema Agent Runs reads (the eve.agent scope, schema version 4, vercel.session_id on Vercel), records usage, and keeps a durable turn's checkpoint. durable-harness adds what the library leaves to its host: where the checkpoints live, how the kernel's drives map onto turns, how each harness describes its model calls, and the export to Vercel.

What changes for sessions

• A turn that goes on in a later invocation (out of time, a crash, a wake, a lane handoff) stays one invoke_agent span. Before, each invocation added a root of its own to the session's trace. Each turn's checkpoint is a stream of the session's run, written in the background and stored before the drive returns, so any process can continue the turn with the same span ids. A tool call that outlives its drive ends in the drive that settles it.
• Content is decided where it is recorded. A session's traceAudience picks a capture decision (public: inputs and outputs; anything else: metadata only), and content a session declines is never written to a span, a checkpoint or an export. agentRuns({ capture }) changes the decision per audience. The per-destination exportPolicy and filteringProcessor are gone: the library does this at the source.
• Agent Runs lists and shows these sessions from their spans alone, so the $eve.* Workflow run tags are no longer written.
• Each model call is a step, and the tool calls its response asks for are under that step, carrying the model's id for the call.

How harnesses report model calls

• The kernel's hook is harness-world/agent-runs, which imports only the library's types, so a drive without agentRuns() registered loads no tracing code. A drive gets ctx.tracing.modelCall(), the library's model call and its step.
• Harness.modelCalls says how a harness reports them. effects (the default) records each model effect from what it logs; self means the harness records each call itself.
• Pi, fx and the AI SDK harness record each call inside their model effect, with the prompt the provider got, its tools, usage and the Gateway's cost. Pi's messages go through harness-world-pi/agent-runs, which pi-durable's storage shares.
• Claude Code, Codex and OpenCode record each request through the Gateway proxy and the VM gateway handler. Requests and responses are read in Anthropic's, OpenAI's and AI Gateway's formats; a long response keeps its start and its end, where the usage is. With tracking on, the proxy reads a request's body before forwarding it.
• pi-durable traces from the commits its storage accepts: a turn per input, a step per generation, a tool call per tool task.
• The demo's libfx lane uses the same hook, and its Event Log drawer links each session to Agent Runs again.

Package

@vercel/agent-runs and @opentelemetry/api are dependencies. OpenTelemetry's SDK stays an optional peer that only /instrumentation/otel loads; it owns its tracer provider and leaves the app's alone. durable-harness now requires Node.js 24, as the library does.

Docs

main's README now points to the docs at vercel.com/docs/agent-harnesses, so this PR changes no README. The Agent Runs pages there (observability, and the API reference) describe the old integration and need to follow: agentRuns({ audience, capture, serviceName, spanProcessors, onError }); traceAudience deciding what a session records; no exportPolicy or filteringProcessor; turns kept across invocations; and Node.js 24.

Testing

• packages/durable-harness/test/agent-runs.test.ts drives sessions through the kernel on world-local. It covers the span tree, the attributes Agent Runs reads, a turn continued across two drives as one span, checkpoints deleted once turns finish, and a private session recording no content.
• packages/cli-check/test/agent-runs.test.ts runs one tool turn of every harness (Pi, pi-durable, fx, AI SDK, Claude Code, Codex, OpenCode) with scripted models; it passes.
• packages/harness-process/src/agent-runs.test.ts covers the request and response readers for each protocol.
• AGENT_RUNS_LIVE=1 against real AI Gateway models passes for Pi, pi-durable, fx, the AI SDK and Codex. Claude Code was skipped: no claude CLI on this machine. OpenCode times out because its model request never starts. That's this machine, not this change: the canary's OpenCode row hangs the same way on a clean main.
• The canary (packages/durable-harness/test/canary.test.ts) searches every span again, with content recorded for every audience. fx, Pi, pi-durable and Codex pass. Claude Code is skipped (no pinned claude binary here), and OpenCode hangs, as it does on main here.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
durable-fx-example	Ready	Preview, v0	Oct 19, 2026 4:13pm UTC
harness-on-world	Ready	Preview, v0	Oct 19, 2026 4:13pm UTC

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	npm/​rollup-plugin-dts@​6.2.3
	npm/​@​vercel/​agent-runs@​0.0.1

View full report

[chadhietala]

Fixed the reported capture-policy issue in 95b24ed.

Both resume-only paths now re-enter the saved turn with the current capture decision before returning handles or settling restored tools. The library narrows the turn and its descendants, so newly private model/tool content is not checkpointed or exported. Missing checkpoints still return no turn.

Regression coverage includes public-to-private/unknown recovery on warm and fresh workers, restricted model/tool content in checkpoints and exports, and memoized kernel tool settlement. Also updated @vercel/agent-runs to 0.0.1.

Validation: 12 tracing/recovery tests plus the scripted pi-durable integration test passed; durable-harness and cli-check typechecks, package build, distribution checks, generic public hygiene, and changeset checks passed. The Linux-only relay caller check was not exercised on macOS.

@vercel-security-reviewer please recheck the finding against this commit.

[pranaygp]

Reviewed with three passes: durability of the tracing core, privacy and packaging, and #11's fx hooks. The design is right: one turn span across invocations, content decided where it's recorded, and nothing loaded without agentRuns(). The three earlier fixes in 0dc07a5 hold.

Requesting changes for:

• Blocker (privacy): a resumed turn keeps the capture decision it started with, in the kernel and in pi-durable. This confirms the security review's finding and is wider than it reported.
• Should fix before merge:
  • the proxy buffers every request body with no cap, even without tracing;
  • drives can hang on unbounded checkpoint I/O;
  • a tracing error can fail a model call;
  • a continued turn is traced under a queued follow-up;
  • a failed checkpoint read looks like no checkpoint;
  • MCP tool tokens are kept in checkpoint streams.

Tests: cli-check and the canary now record content for every audience, so apart from one scripted check, no harness path tests a private or unknown session any more. Could expectToolTurn run each harness again with the default capture and unknown audience, and assert the marker is in no span and no _agent-runs_ stream line?

CI's test failure here was the harness-sandbox relay race, which this PR doesn't touch; I re-ran it.

[vercel-security-reviewer]

Review complete.

[pranaygp]

@chadhietala thanks, 95b24ed fixes the resume capture blocker in both paths. I've marked those threads.

To get #10 and #11 in for launch without colliding on your branches, I'm taking the rest of the review on stacked branches, each with a PR into yours:

• pgp/agent-runs-fixes → chad/agent-runs-library: the proxy reading bodies without tracing; bounded tracing I/O; the model path; the turn chosen when a follow-up is queued; failed checkpoint reads; tool definitions in checkpoints; private-audience coverage in cli-check.
• pgp/agent-runs-fx-fixes → chad/agent-runs-fx-hooks: handoff and fenced turns, interrupted, session eviction, executions, outcome tests.

Keep pushing to your branches as you like; I'll merge yours into mine. If you've already started on any of these, reply here and I'll drop it.

🤖 Posted by Claude Code

[chadhietala]

Addressed the review in ff4eb4b, d9e4484, and 3963827 (privacy recovery was already fixed in 95b24ed).

• Bounded per-run checkpoint I/O, strict failed-read handling, tail recovery, bounded idle caches, and quarantine of timed-out writes until the actual World write settles.
• Stream untraced/oversized proxy requests, isolate tracing errors from model/tool outcomes, and keep continued turns ahead of queued follow-ups.
• Project tool definitions/message metadata before capture; make agent-runs@0.0.1 an optional peer with inlined declarations and preserve Node 22 for ordinary consumers.
• All seven scripted harnesses now run with both content capture and default unknown-audience capture; metadata-only checks inspect exported spans AND checkpoint stream bytes. Added an actual public-to-private pi-durable reopen test.

Validation: 39 tracing/recovery/packaging tests, the 14-case scripted matrix, 32 process gateway/model tests, eight workspace typechecks, package build, declaration resolution with agent-runs disabled, check-dist, and generic public hygiene passed. The broader durable-harness suite timed out; isolating public-sandbox.test.ts shows four OpenCode cases failing because the installed @opencode/core cannot resolve @effect/platform-node (same on both branches). Linux relay caller validation remains CI-only.

The deletion-only worktree was not touched. Requesting re-review.

[pranaygp]

Thanks for ff4eb4b, d9e4484 and 3963827. They fix every item from my last review; I checked each against the code, and the affected tests and the typecheck pass at 3963827.

Two new problems, both reproduced. Each makes a continued or completed turn export its spans twice, with the same span id. Neither leaks content. The first hits the main cross-invocation path (every drive that runs out of its budget), so I'd fix both before merging. Details inline.

NIT (outside this diff): harness-process's README ModelSteps example (README.md:398) shows gatewayProxy({ credential, token, track: steps.request }) without captureRequest. Code copied from it now records model steps with no request messages. It could add captureRequest: ctx.tracing !== undefined, as the OpenCode and reference harnesses do.

🤖 Posted by Claude Code

[chadhietala]

Fixed both new duplicate-export findings in cf4fb1a and updated the ModelSteps README example with captureRequest. Both new regressions were reproduced before the fix.

Validation: 43 tracing/recovery/packaging tests and all 14 scripted harness capture-policy cases passed, plus package typecheck, build, check-dist, generic public hygiene, and changeset checks. The soft-deadline regression verifies unique span IDs across fresh-process recovery; the slow-write regression verifies the durable finished marker prevents a second turn export.

Per the requested release-note policy, #10 now adds exactly one changeset. #11 includes this base fix and likewise adds exactly one fx changeset. Requesting re-review.

[pranaygp]

@chadhietala re-reviewed at 1f3279d / b778a66: both checkpoint fixes are right. The four new tests fail on 3963827's code and pass on yours. The parked-tool fix closes the cut-off-call gap on #11 too. Approving.

One change on your branches: main fixed the same flaky OpenCode test in #15 (dda5ef2), so 2f07eeb (here) and c30b6e3 (#11) take main's version of harness-opencode/src/harness.test.ts. That was #10's only conflict. Once CI is green, I'll merge the stack (#10, then #11) and cut the next beta.

🤖 Posted by Claude Code