[David Elner (delner)]

Based on #2725

Why

LiveKit voice traces now follow audio schema v3. Some of the major changes from this:

• Speech and user turns carry audio.selections in reference to the session's timeline, so a span points at its exact place in the recording. This is recorded even when audio capture is off.
• Attribution for sources, participants (e.g. a caller), and speakers (individual voices) including support for diarization (identifying which text belonged to what speaker.)
• Links between when operations, when possible. e.g. A model response links to the transcripts in its context, a tool to the response that requested it, and played speech to the synthesis and model response that generated it.

Example

A two-caller order call with a diarizing STT (from the e2e fixture):

session          audio.schema.version: 3.0.0 · sources · speakers · segments · timeline
├── user_turn    input: "Hi. I'm calling with my partner, … Where is order demo one zero four two? …"
│   ├── user_speech ×4                                        audio.selections
│   ├── stt  "Hi. I'm calling with my partner,"               speaker-1
│   ├── stt  "who has a question about our order."            speaker-1
│   ├── stt  "Where is order demo one zero four two?"         speaker-2
│   └── …
└── agent_turn   input: (the transcript above) · turn.reply_to → user_turn
    ├── llm_response   uses_context → each stt
    ├── lookup_order   requested_by → llm_response
    ├── llm_response   uses_context → each stt · consumes_tool_result → lookup_order
    ├── tts            generated_by → llm_response
    └── agent_speech   "Your order … is scheduled for delivery on Friday."
                       audio.selections · generated_by → tts

Session metadata, illustrated for a call with a room participant:

{
  "audio.schema.version": "3.0.0",
  "audio.sources": { "session-input": { "label": "Agent input audio" }, "agent-output": { "label": "Agent audio" } },
  "audio.speakers": { "speaker-1": { "type": "user", "attributes": { "contrib.livekit.speaker_id": "0" } } },
  "audio.participants": { "participant-1": { "type": "user", "attributes": { "contrib.livekit.participant_id": "PA_x7Kq", "contrib.livekit.participant_kind": "SIP" } } },
  "audio.segments": { "part-0": { "state": "ready", "start_offset_ms": 0, "duration_ms": 30000, "attachment": { "ref": "/input/audio/part-0" } } }
}

Design notes

• Prefer keys and IDs native to the framework rather than heuristics. Replies match turns by LiveKit's identities, not timestamps. An STT span runs from the first result to the final, under the turn it began in. Heard text comes from the sink's synchronized transcript.
• Lean on more reliable mechanics for timing. Agent playout is timed by the sink, even after an interruption. User speech is selected only with a local VAD, since provider speech events arrive late.
• Identities can only be added when LiveKit knows them. Without rooms, diarization, or PII permitted, some forms of identification are not always possible.
• New fields follow fix(livekit): Respect LiveKit's PII and project redaction settings #2724's PII gate.