Repository navigation
fix(livekit): Respect LiveKit's PII and project redaction settings - #2724
Merged
Merged
Conversation
…ario exits The scenario exits with process.exit(0) after a successful run, which skips its finally block, so websocket cassettes were never written when recording and never checked for full consumption in replay. Close the cassette first. The cassette also takes the provider's upstream and authorization, so it can proxy LiveKit Inference as well as OpenAI.
LiveKit omits PII from its own telemetry when LIVEKIT_TELEMETRY_ALLOW_PII is falsy, when the job's LiveKit Cloud project enables redaction, or when a span carries lk.redaction.enabled. The plugin reads native attributes before LiveKit's redaction runs, so it traced transcripts, prompts and tool payloads regardless. It now applies LiveKit's own rule (its pii key segment, gen_ai content keys, prompt variables and message events, with exception details replaced) whenever LiveKit's configuration disallows PII, and skips the text LiveKit reports through events: transcripts, assistant and heard text, and tool definitions. PII values are still read in memory to resolve links between spans, which hold only span IDs, so the call stays connected. The job's redaction setting is read through LiveKit's getJobContext, which auto-instrumentation hooks and wrapLiveKit hands to the plugin. A span's redaction flag applies to its whole session, so observations without a native span of their own are covered too. Structure, timing, models, usage, identities and opt-in audio are kept.
David Elner (delner)
changed the title
This was referenced Nov 3, 2026
Luca Forstner (lforst)
approved these changes
Nov 3, 2026
Luca Forstner (lforst)
pushed a commit
that referenced
this pull request
Nov 3, 2026
…2725) Based on #2724 ## Why Every voice integration has to describe its session the same way under audio schema v3: sources, a shared timeline, participants, speakers, recorded segments, and each span's place on that timeline. `AudioSessionRecorder` owns that shape once, so integrations only translate their framework's events into calls. - **Provides common implementation for session shape** to reduce duplication per integration. - **Provides session-local IDs** (`participant-N`, `speaker-N`) derived from native values. - **With audio capture disabled, it can still describes the session**, so timing and identities can still be relayed on the trace. In sum, this enables LiveKit's move to schema v3 (next PR), and future voice integrations. ## Example How an integration uses it: ```ts const audio = new AudioSessionRecorder({ origin: Date.now(), sources: { "session-input": { label: "Agent input audio" }, "agent-output": { label: "Agent audio" }, }, audio: { sampleRate: 48_000, segmentBytes }, // omit to describe without recording async onFile({ id, data, mimeType, metadata }) { // Upload `data` as input.audio[id], then log it with its audio.segments entry. session.log({ input: { audio: { [id]: await upload(data, mimeType) } }, metadata }); }, }); session.log({ metadata: audio.session() }); // schema version, sources, timeline const { id, added } = audio.participant("PA_x7Kq", { type: "user", attributes: { "contrib.livekit.participant_id": "PA_x7Kq" }, }); if (added) session.log({ metadata: added }); // audio.participants, first time only span.log({ metadata: { "audio.participant_ids": [id] } }); audio.record("session-input", pcm, { sampleRate: 16_000, channelCount: 1, offsetMs }); span.log({ metadata: { "audio.selections": audioSelections([{ start, end }]) } }); await audio.finish(); ``` ## Design notes ``` integration ──calls──► AudioSessionRecorder ──returns──► metadata to log │ └─ AudioRecorder (when recording) ─► onFile(segment + entry) ``` - The `AudioSessionRecorder` is responsible for managing the session, resolving references, etc but it is not responsible for instrumentation or trace data itself... it merely returns metadata that instrumentation can log. - It tracks commonly referenced objects like `participant` and `speaker`, and helps spans map to a common, PII-less ID. (e.g. `participant-1`)
Luca Forstner (lforst)
pushed a commit
that referenced
this pull request
Nov 3, 2026
Based on #2725 ## Why LiveKit voice traces now follow audio schema v3. Some of the major changes from this: - Speech and user turns carry `audio.selections` in reference to the session's timeline, so a span points at its exact place in the recording. This is recorded even when audio capture is off. - Attribution for sources, participants (e.g. a caller), and speakers (individual voices) including support for diarization (identifying which text belonged to what speaker.) - Links between when operations, when possible. e.g. A model response links to the transcripts in its context, a tool to the response that requested it, and played speech to the synthesis and model response that generated it. ## Example A two-caller order call with a diarizing STT (from the e2e fixture): ``` session audio.schema.version: 3.0.0 · sources · speakers · segments · timeline ├── user_turn input: "Hi. I'm calling with my partner, … Where is order demo one zero four two? …" │ ├── user_speech ×4 audio.selections │ ├── stt "Hi. I'm calling with my partner," speaker-1 │ ├── stt "who has a question about our order." speaker-1 │ ├── stt "Where is order demo one zero four two?" speaker-2 │ └── … └── agent_turn input: (the transcript above) · turn.reply_to → user_turn ├── llm_response uses_context → each stt ├── lookup_order requested_by → llm_response ├── llm_response uses_context → each stt · consumes_tool_result → lookup_order ├── tts generated_by → llm_response └── agent_speech "Your order … is scheduled for delivery on Friday." audio.selections · generated_by → tts ``` Session metadata, illustrated for a call with a room participant: ```json { "audio.schema.version": "3.0.0", "audio.sources": { "session-input": { "label": "Agent input audio" }, "agent-output": { "label": "Agent audio" } }, "audio.speakers": { "speaker-1": { "type": "user", "attributes": { "contrib.livekit.speaker_id": "0" } } }, "audio.participants": { "participant-1": { "type": "user", "attributes": { "contrib.livekit.participant_id": "PA_x7Kq", "contrib.livekit.participant_kind": "SIP" } } }, "audio.segments": { "part-0": { "state": "ready", "start_offset_ms": 0, "duration_ms": 30000, "attachment": { "ref": "/input/audio/part-0" } } } } ``` ## Design notes - Prefer keys and IDs native to the framework rather than heuristics. Replies match turns by LiveKit's identities, not timestamps. An STT span runs from the first result to the final, under the turn it began in. Heard text comes from the sink's synchronized transcript. - Lean on more reliable mechanics for timing. Agent playout is timed by the sink, even after an interruption. User speech is selected only with a local VAD, since provider speech events arrive late. - Identities can only be added when LiveKit knows them. Without rooms, diarization, or PII permitted, some forms of identification are not always possible. - New fields follow #2724's PII gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
LiveKit can be configured to withhold PII from its telemetry, but Braintrust's LiveKit traces ignored that setting. The plugin reads LiveKit's span attributes and events before LiveKit's own redaction runs, so traces still carried transcripts, prompts, model responses, tool arguments and results, and exception details.
LiveKit withholds PII when any of these apply:
LIVEKIT_TELEMETRY_ALLOW_PIIis falsy (0,false,no,off)lk.redaction.enabledSolution
When LiveKit's configuration withholds PII, the plugin applies LiveKit's own redaction rule before anything is logged.
When PII is disallowed it omits:
lk.*pii*attributes, gen_ai content attributes, prompt variables and message eventsexception details redacted)When PII is disallowed it retains:
BRAINTRUSTflags).NOTE:
setTracerProvider(..., { allowPii: false }), is not covered because LiveKit doesn't expose it to the plugin.Separately, this also fixes the LiveKit e2e websocket cassette being saved after the scenario exited.