Merge latest main into v4 (#2680)

Co-authored-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
Co-authored-by: Andrew Kent <31153155+realark@users.noreply.github.com>
Co-authored-by: Abhijeet Prasad <abhijeet@braintrustdata.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: alexcastgonz <82875882+pcbeingused333@users.noreply.github.com>
Co-authored-by: theanecdote <sungeun.jp@icloud.com>
Co-authored-by: Amp <amp@ampcode.com>
Co-authored-by: David Elner <david.elner@braintrustdata.com>

Author: 10 people

.agents/skills/create-changeset/SKILL.md

@@ -6,7 +6,18 @@ description: Create or update Changesets for this repository with a Conventional
 # Create a Changeset
 
 Inspect the diff to identify every publishable package affected and choose the appropriate semantic version bump for each package.
-Use `pnpm changeset` when creating the changeset unless the existing task requires editing a specific changeset file.
+Create the changeset by writing a new Markdown file under `.changeset/` directly, with a short kebab-case filename such as `.changeset/fix-queued-login-race.md`.
+
+```markdown
+---
+"braintrust": patch
+"@braintrust/otel": minor
+---
+
+fix: Prevent conflicting queued logins from replacing active credentials
+```
+
+List each affected package by its `package.json` name with `major`, `minor`, or `patch`.
 
 ## Summary
 

.agents/skills/instrumentation/SKILL.md

@@ -1,23 +1,23 @@
 ---
 name: instrumentation
-description: Add or update Braintrust SDK instrumentation. Use when working on instrumentation of any kind - like wrappers, auto-instrumentation configs, tracing channels, provider plugins, vendored SDK typings, or instrumentation-specific tests.
+description: Add or update Braintrust SDK instrumentation. Use when working on instrumentation of any kind - like wrappers, auto-instrumentation configs, invocation hooks, provider plugins, vendored SDK typings, or instrumentation-specific tests.
 ---
 
 # Instrumentation Rules
 
 Read first based on the task:
 
-- `js/src/instrumentation/README.md` for plugin and tracing-channel architecture
-- Closest file in `js/src/instrumentation/core/` when changing shared channel semantics
+- `js/src/instrumentation/README.md` for wrapping and tracing architecture
+- Closest file in `js/src/instrumentation/core/` when changing shared invocation semantics
 - Closest file in `js/src/instrumentation/plugins/` when changing provider-specific extraction or span mapping
 - Closest file in `js/src/wrappers/` when manual wrappers and auto-instrumentation need to stay aligned
 - Closest test in `js/tests/auto-instrumentations/` when changing hook, loader, bundler, or transform behavior
 - Closest e2e scenario in `e2e/scenarios/*instrumentation*` or `e2e/scenarios/*node-hook*` when the user-visible trace contract changes
 
 Map the change before editing:
 
-- `js/src/instrumentation/core/` - tracing-channel helpers, stream patching, shared types
-- `js/src/instrumentation/plugins/` - provider-specific channel subscriptions and event-to-span conversion
+- `js/src/instrumentation/core/` - invocation definitions, independent tracing helpers, stream patching, shared types
+- `js/src/instrumentation/plugins/` - explicit interceptor registration and provider-specific tracing functions
 - `js/src/wrappers/` - manual instrumentation entrypoints that should mirror the same logical contracts
 - `js/src/auto-instrumentations/` - loader and bundler instrumentation config
 - `js/tests/auto-instrumentations/` - functional coverage for transformed code
@@ -27,10 +27,22 @@ Map the change before editing:
 - Inputs are untrusted: treat args, results, events, headers, and metadata as hostile. Prototype pollution is a concrete risk here. Avoid unsafe property access patterns, prototype-sensitive operations, and unnecessary mutation of third-party objects.
 - Support both auto-instrumentation and manual instrumentation. Auto-instrumentation does not cover every environment, loader, or framework.
 - For orchestrion auto-instrumentation, prefer targeting public API functions. Instrumenting internal helpers is more likely to break across library versions.
-- Auto and manual paths should share logic through the same typed channel. For new and migrated instrumentation, prefer `invoke` in manual wrappers and `intercept` in provider plugins so the target can be scoped with `AsyncLocalStorage.run()` and its arguments, receiver, or output can be patched. Keep tracing-style hooks only as the compatibility path for instrumentation that has not migrated yet. Manual wrappers should not directly emit observability data.
+- Braintrust's vendored Orchestrion implementation is purely internal to the SDK.
+  Downstream packages must not depend on its APIs, config schema, generated helpers, or compatibility with upstream Orchestrion.
+  Do not expose these internals through SDK entrypoints or public configuration options.
+  Update the transformer and its SDK callers together; remove unused upstream features and compatibility scaffolding.
+  Preserve instrumented application behavior, coordination between SDK copies, and upstream license notices.
+- Keep wrapping and tracing separate, including internal APIs.
+  Define typed invocation hooks with `defineInterceptor`; `intercept` and `invoke` only compose and execute wrappers.
+  Definitions describe call arguments, return values, and opaque additional data, without span provenance or tracing methods.
+  The wrapping runtime must work without SDK initialization and have no dependency on spans, logging, or tracing lifecycle events.
+  Put span creation, context propagation, and finalization in separate tracing functions.
+  Plugins explicitly register those functions through `intercept`; shared tracing helpers accept callables and tracing configuration, never hooks or registration responsibilities.
+  Do not add combined helpers such as `traceInvocation` or `interceptAndTrace`, or retain a tracing-event compatibility path.
+  Manual wrappers and generated wrappers use the same hook through `invoke` and do not directly emit observability data.
 - Reuse shared repo utilities before introducing local helpers. Check `js/util/index.ts`, neighboring instrumentation files, and existing plugins/wrappers for utilities like `isObject`, merge helpers, and sanitizers before adding ad hoc replacements.
 - If a public instrumentation surface changes, check whether the export surface also needs updates in `js/src/instrumentation/index.ts` or `js/src/exports.ts`.
-- Preserve async context propagation. Changes around tracing channels, stream patching, or loader hooks must keep the current span context across awaits and stream consumption.
+- Preserve async context propagation. Changes around invocation hooks, stream patching, or loader hooks must keep the current span context across awaits and stream consumption.
 - Maintain isomorphic behavior. Node and browser/bundled paths must use compatible channel implementations and avoid channel-registry mismatches.
 - Setup, teardown, and patching must be idempotent. Enabling twice, disabling twice, or applying a patch twice should remain safe.
 - Promise/stream behavior must be preserved. Patches need to keep subclass/helper semantics intact.
@@ -41,10 +53,18 @@ Map the change before editing:
 - Log only the useful surface. Prefer narrow, stable payloads over dumping full request/response objects; exclude redundant or overly large data when possible.
 - Automatic capture of inline multimodal bytes as Braintrust attachments must be opt-in.
   Guard attachment creation and binary observation with `BRAINTRUST_AUTO_CAPTURE_ATTACHMENTS`, and enable them only when its value is `1` or `true`.
+  Audio/video recordings may contain personal data and are separately controlled with their own explicit flags (`BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS`, `BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS`): enabling the shared flag for other media never records audio/video without these being enabled.
   Preserve useful text, metadata, metrics, and remote references when attachment capture is disabled, and omit inline bytes from logged payloads.
+- Segmentable streaming media must use the shared `configureInstrumentation({ streamingAttachmentBufferBytes })` setting instead of provider-specific chunk-size or time-based controls.
+  It defaults to 4 MiB per logical media stream and counts binary attachment payload bytes after decoding and format conversion, including any required padding or silence.
+  Flush near the threshold at valid media boundaries and flush remaining valid content when the recording ends.
+  Preserve ordering, format metadata, and timestamps needed to interpret the attachments.
+  Keep whole-file responses intact, even if they exceed the threshold; never arbitrarily split encoded images, audio, or video.
+  This is a flush threshold, not a total memory limit or attachment-capture opt-in.
+  Preserve application stream behavior and existing capture consent; keep any capture safety ceiling internal and distinct from the threshold.
 - Do not modify package READMEs during instrumentation work unless the user specifically requests it or the change corrects outdated information.
 - We want to limit our instrumentation to operations that are relevant for AI generations and operations (LLMs, embeddings, media generation, ...). Things like creating entities on platforms (CRUD for Workflows of Agent entities) is irrelevant to us.
-- When building instrumentation, we should always have a vendored type/interface for what we are wrapping. The type or interface should not be larger than what is relevant to the instrumentation. The type or interface should be used for typing tracing channels and also should be used to assert the type on whatever is passed into wrappers as soon as the wrapper has verified that the passed in value is plausibly what should be wrapped.
+- When building instrumentation, we should always have a vendored type/interface for what we are wrapping. The type or interface should not be larger than what is relevant to the instrumentation. The type or interface should be used for typing invocation hooks and also should be used to assert the type on whatever is passed into wrappers as soon as the wrapper has verified that the passed in value is plausibly what should be wrapped.
 
 ## Process
 

.changeset/livekit-audio-capture-flags.md

@@ -0,0 +1,7 @@
+---
+"braintrust": minor
+---
+
+feat(livekit): Record caller and agent audio with explicit per-direction flags
+
+Set `BRAINTRUST_CAPTURE_USER_AUDIO_ATTACHMENTS` and `BRAINTRUST_CAPTURE_AGENT_AUDIO_ATTACHMENTS` to record each side of a call.

.changeset/livekit-close-waits-for-recording.md

@@ -0,0 +1,5 @@
+---
+"braintrust": patch
+---
+
+fix(livekit): Finish uploading recordings before `session.close()` resolves, so they are not lost when the job process exits

AGENTS.md

@@ -46,8 +46,22 @@ Keep public exports minimal. Generally, export only the requested runtime APIs a
 
 ## Instrumentation
 
+Our vendored Orchestrion implementation is purely internal to the Braintrust SDK.
+Downstream packages must not depend on its APIs, config schema, generated helpers, or compatibility with upstream Orchestrion.
+Do not expose these internals through SDK entrypoints or public configuration options.
+Change the transformer and its SDK callers together, and remove unused upstream features instead of retaining compatibility scaffolding.
+Preserve instrumented application behavior, coordination between SDK copies, and upstream license notices.
+
 Use the normal Orchestrion config plus plugin/channel path by default. Special-case source patches should be rare exceptions only when the target SDK cannot be instrumented through the standard transformer path, and the reason should be documented next to the patch.
 
+API wrapping and span instrumentation are separate concepts.
+Use `defineInterceptor` to define invocation hooks and `intercept`/`invoke` exclusively as generic wrapping machinery.
+Wrapping definitions and runtime code must not depend on spans, tracing lifecycle events, provenance, or SDK initialization.
+Keep span creation, context propagation, and finalization in separate tracing functions.
+Provider plugins explicitly register those functions through interceptors; tracing helpers accept callables and tracing configuration, never hooks or registration responsibilities.
+Do not introduce combined APIs such as `traceInvocation` or `interceptAndTrace`, including internal convenience APIs.
+Manual wrappers and generated wrappers only invoke hooks.
+
 Instrumentation patches generally do not need to be removed during teardown. Prefer leaving behavior-preserving patches installed when they are idempotent; do not add unpatching machinery by default.
 
 Span names should generally remain stable across calls and versions. Do not include dynamic values such as model names in span names; record those values in metadata instead.
@@ -61,11 +75,17 @@ Generally, do not add tests for build or linting code or their configuration. Va
 **From `js/` directory:**
 
 ```bash
-pnpm test                         # Core vitest suite (excludes wrappers)
-pnpm test -- -t "test name"       # Filter by test name
-pnpm run test:checks              # Hermetic tests (core + vitest wrapper)
+pnpm test                                    # Core vitest suite (excludes wrappers)
+pnpm exec vitest run src/foo.test.ts         # Run a single test file
+pnpm exec vitest run src/foo.test.ts -t "x"  # Filter by test name within a file
+pnpm run test:checks                         # Hermetic tests (core + vitest wrapper)
+pnpm run check:typings                       # Typecheck source and test files
 ```
 
+Do not pass filters through `pnpm test -- ...`; the `--` makes Vitest ignore them and run the full suite.
+
+Vitest does not typecheck. Before opening or updating a pull request, run `pnpm run check:typings` in each package you touched (or from the repo root) so type errors in source or test files are caught before CI.
+
 **E2E tests (`e2e/`):**
 
 Each scenario runs the SDK in a subprocess against a mock Braintrust server and snapshots the results. No API keys required for replay; recording needs provider keys.

PUBLISHING.md

@@ -4,7 +4,7 @@ All package publishing happens in GitHub Actions. Do not publish from your local
 
 ## Short Version
 
-- If your PR changes a publishable package, run `pnpm changeset` and commit the generated `.changeset/*.md` file.
+- If your PR changes a publishable package, add a `.changeset/*.md` file to it.
 - Merge the PR into `main`.
 - For a stable release:
   - Run **Prepare Release** on `main`.
@@ -21,19 +21,7 @@ All package publishing happens in GitHub Actions. Do not publish from your local
 
 If your PR changes a publishable package, CI expects a changeset unless you explicitly skip it.
 
-Create one with:
-
-```bash
-pnpm changeset
-```
-
-That creates a `.changeset/*.md` file. Commit it with your PR.
-
-Use:
-
-- `patch` for fixes and non-breaking maintenance changes
-- `minor` for new backwards-compatible features
-- `major` for breaking changes
+To add one, have an agent follow [`.agents/skills/create-changeset/SKILL.md`](.agents/skills/create-changeset/SKILL.md), then commit the generated `.changeset/*.md` file with your PR.
 
 If no release is intended, bypass the check with either:
 

dev-packages/seinfeld/src/recorder.ts

@@ -64,6 +64,7 @@ interface CassetteContext {
   ignoredRequests: IgnoredRequestMatcher[] | undefined;
   streamingRequests: StreamingRequestMatcher[] | undefined;
   threshold: number | false;
+  inlineRequestBodies: boolean;
   onMiss: ((req: RecordedRequest) => void) | undefined;
 
   // Per-run mutable state (reset on each start)
@@ -106,11 +107,17 @@ async function handleCassetteRequest(
 
   switch (ctx.mode) {
     case "replay": {
-      const recorded = await recordRequest(request, ctx.threshold);
+      const recorded = await recordRequest(
+        request,
+        ctx.inlineRequestBodies ? false : ctx.threshold,
+      );
       return handleReplay(ctx, recorded);
     }
     case "record": {
-      const draft = await recordRequestDraft(request, ctx.threshold);
+      const draft = await recordRequestDraft(
+        request,
+        ctx.inlineRequestBodies ? false : ctx.threshold,
+      );
       return handleRecord(ctx, getRealResponse, draft);
     }
     case "passthrough":
@@ -449,6 +456,8 @@ export interface CassetteServerOptions {
   store: CassetteStore;
   /** Filter spec (matching-only normalization). */
   filters?: FilterSpec;
+  /** Keep large request bodies inline so matching filters can normalize their bytes. */
+  inlineRequestBodies?: boolean;
   /** Redaction spec for what gets persisted to disk. Defaults to paranoid. */
   redact?: RedactionSpec;
   /**
@@ -496,6 +505,7 @@ function createContext(options: CassetteServerOptions): CassetteContext {
     ignoredRequests: options.ignoredRequests,
     streamingRequests: options.streamingRequests,
     threshold: DEFAULT_EXTERNAL_BLOB_THRESHOLD,
+    inlineRequestBodies: options.inlineRequestBodies ?? false,
     onMiss: options.onMiss,
     candidates: [],
     callCounts: new Map(),
@@ -797,15 +807,25 @@ async function writeFetchResponse(
 
   const reader = response.body.getReader();
   try {
-    for (;;) {
+    while (!res.destroyed) {
       const { done, value } = await reader.read();
       if (done) break;
       if (!res.write(Buffer.from(value))) {
-        await new Promise<void>((resolve) => res.once("drain", resolve));
+        // A client that disconnects mid-stream never drains.
+        await new Promise<void>((resolve) => {
+          const resume = () => {
+            res.off("drain", resume);
+            res.off("close", resume);
+            resolve();
+          };
+          res.on("drain", resume);
+          res.on("close", resume);
+        });
       }
     }
   } finally {
     res.end();
-    reader.releaseLock();
+    if (res.destroyed) await reader.cancel();
+    else reader.releaseLock();
   }
 }

e2e/README.md

@@ -183,7 +183,19 @@ pnpm --filter=@braintrust/js-e2e-tests run test:e2e
 
 If a scenario records but later replay fails because of volatile fields in the request body (e.g. AI-SDK's generated message ids), add or update `<scenario-dir>/cassette-filter.mjs` for that scenario, then re-record.
 
-After any successful record run, stale cassette variants are cleaned only inside scenarios that emitted cassette run-context records during that run. With explicit scenario names, cleanup is limited to those scenarios; with a full record run, cleanup applies to every observed cassette-backed scenario. Per-cassette blob sidecar cleanup still happens in the cassette file store when each cassette is saved.
+After a successful record run, unused cassette variants and their blob directories are cleaned only for scenarios whose test files and tests all passed. Cleanup uses the cassette keys actually selected by the harness, including overrides and opt-outs. Scenarios with skipped tests, missing test files, or missing usage records are left untouched. Per-cassette blob sidecar cleanup still happens in the cassette file store when each cassette is saved.
+
+### Checking unused cassettes without re-recording
+
+```bash
+pnpm run test:e2e:cassettes
+pnpm run test:e2e:cassettes -- cloudflare-think-instrumentation
+pnpm run test:e2e:cassettes -- --clean-cassettes cloudflare-think-instrumentation
+```
+
+These commands run a fresh replay and compare the cassette files on disk with actual usage. The default check leaves files untouched and exits nonzero when it finds unused variants. `--clean-cassettes` removes unused JSON files and their `.cassette.blobs` directories after the tests pass.
+
+Only whole scenarios can be selected; test filters, file paths, sharding, and snapshot updates are rejected. A scenario with skipped tests, missing test files, or missing usage records is reported as not checked and makes the command exit nonzero. Unselected scenarios and directories with no test files are not checked. Use `BRAINTRUST_E2E_RUN_CONTEXT_DIR` to retain the run records and test report for the trace summary.
 
 ### In-scope scenarios