Skip to content

Low, constant latency audio path (playout buffer with drift correction) - #186

Merged
teamclouday merged 3 commits into
teamclouday:mainfrom
emdaitaj:low-latency-playout
Oct 19, 2026
Merged

teamclouday merged 3 commits into
teamclouday:mainfrom
emdaitaj:low-latency-playout

Conversation

@emdaitaj

Copy link
Copy Markdown
Contributor

Summary

End-to-end latency was around 1 s and drifted. This PR keeps it low and constant: if two sounds reach the phone mic 204.53 ms apart, apps listening on the virtual mic also hear them 204.53 ms apart (measured within 0.014 ms, below). That matters for real-time analyzers and other measurement tools, and it also helps calls and recording.

What caused the delay

  • PC: audio went into a 1 s ring buffer with nothing controlling its level. Network bursts and the small clock difference between phone and sound card filled it up, and the latency stayed there.
  • Android, TCP: Nagle's algorithm was on, and the length prefix and payload were written separately, so packets could wait on an ack.
  • Android, capture: each read took a whole minBufferSize (often 20–40 ms) before anything was sent.
  • Android, Wi-Fi: power saving delayed packets by tens to hundreds of ms.
  • PC, UI: every packet's wave update awaited the UI channel, so a busy UI held back the network reads.
  • PC, UDP: late packets were logged as dropped but still played, and lost packets shifted the timeline.

Changes

PC (RustApp)

New audio/playout.rs sits between the network side and the cpal output stream:

  • Fixed jitter margin. Playback starts once the margin is buffered. For the first 3 s it watches how low the buffer actually goes, and holds playback once more if needed. Phones that deliver audio in 10–20 ms bursts still get the full margin.
  • Clock drift correction. A band-limited polyphase resampler (64 taps, flat within 0.001 dB to 20 kHz at 48 kHz, 0.67 ms fixed delay) cancels the drift.
  • How drift is measured. It's the ratio of two long line fits: frames received over time, fitted on the network side using the last packet of each burst, and frames played over time. The buffer level isn't used because it jumps with every packet and callback. After 60 s the latency is locked, with only a very weak correction on top.
  • Late audio. It's replaced by silence, then skipped when it arrives, so the audio after a hiccup keeps the same latency. After a 500 ms stall it starts over, and a large backlog jumps back to the margin.
  • UDP. Late or duplicate packets are dropped, and lost packets are filled with the right amount of silence.
  • UI. Audio wave updates use try_send, so they never block the network task.
  • Setting. A new "Jitter buffer" slider in the connection section (target_latency_ms, default 30 ms). The stream is rebuilt when the slider is released.
  • Ring buffer. It now carries interleaved f32, and is converted to the device format in the callback.

Android

  • TCP_NODELAY, and one write per packet for TCP and USB.
  • 5 ms packets, read on a THREAD_PRIORITY_URGENT_AUDIO thread. The 500 ms record buffer is kept drained, so it adds no latency; if the device refuses that size it falls back to the minimum.
  • While muted, silence is streamed instead of stopping, so no stale audio piles up and the PC doesn't have to re-buffer.
  • Wi-Fi high-perf / low-latency locks while streaming over Wi-Fi or UDP (WAKE_LOCK permission).
  • New "Unprocessed (forced)" audio source. Some devices (e.g. Samsung) don't report PROPERTY_SUPPORT_AUDIO_SOURCE_UNPROCESSED but still accept the unprocessed source.

Measurements

Galaxy S24 Ultra (Android 16), ADB mode, 48 kHz stereo i24, jitter buffer 20 ms, Windows 11 with VB-Cable. PC speakers played beeps every 204.53 ms; the phone picked them up and the beeps were recorded from CABLE Output. Each measurement took 40 s, after the latency lock.

before after
end-to-end latency ~1 s, varying 244.6 ms, spread 0.097 ms
error of each 204.53 ms interval n/a 0.0048 ms rms, 0.0142 ms worst
timeline wander n/a 0.0073 ms rms, 0.036 ms peak to peak
  • Where the 244 ms goes: about 63 ms from the speaker through the phone mic to arrival on the PC (this phone delivers mic audio in 20 ms bursts), ~25–45 ms in the playout buffer, and 103 ms in VB-Cable alone (measured with no phone involved).
  • Simulated jitter and drift: unit tests in playout.rs add clock drift from −200 to +300 ppm, 15 ms of jitter, and a 150 ms stall. They check that latency stays within 0.03 ms and each interval within 0.005 ms, and that the stall doesn't shift the timeline.

Testing

  • cargo test --release passes; cargo check --benches passes.
  • ./gradlew assembleRelease passes.
  • Tested on the phone above in ADB mode. I didn't test Wi-Fi/UDP/USB on hardware; they share the same playout path.

Notes

  • On the S24 Ultra, "Unprocessed (forced)" records the same signal as Voice Recognition, 12 dB quieter, with no extra headroom. It may help on devices where UNPROCESSED is really raw.
  • If a phone itself inserts audio (seen on the S24 when its own media playback starts), latency steps up by ~10 ms and is only corrected slowly. This doesn't happen when the sound source is external.

🤖 Generated with Claude Code

emdaitaj and others added 3 commits October 19, 2026 14:19
The PC kept up to 1 s of audio in a plain FIFO with nothing holding its
level, so network bursts and the phone/PC clock difference filled it up
and the latency stayed there. Timing also jittered from TCP Nagle,
large capture chunks, wifi power saving and the UI back pressuring the
network task.

PC:
- new playout buffer feeding the sound card: holds a fixed jitter
  margin (configurable, default 30 ms), cancels the clock drift with a
  band limited resampler (flat to 20 kHz), and replaces late audio with
  silence then skips it, so the timeline never shifts
- the drift comes from line fits of frames received (network side, last
  packet of each burst) and frames played over time, not from the
  jumpy buffer level
- after start the lowest buffer level is probed for 3 s and playback is
  held back once if needed, so bursty phones get the full margin
- UDP: late packets are dropped and lost ones are filled with silence
- audio wave updates no longer block the network task
- "Jitter buffer" setting in the connection section

Android:
- TCP_NODELAY and one write per packet (tcp and usb)
- 5 ms packets, read on an urgent audio priority thread, with a large
  record buffer that is kept drained (falls back to the minimum size)
- keep streaming silence while muted instead of stopping
- wifi high perf / low latency lock while streaming over wifi

Measured on a Galaxy S24 Ultra over ADB, beeps every 204.53 ms played
by the PC speakers and recorded on VB-Cable: interval error 0.0025 ms
rms, 0.0078 ms worst; end to end 232 ms (103 ms of it is VB-Cable).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- the overflow jump keeps the margin added by the startup probe,
  instead of undoing it
- bursts are detected relative to the packet length, so slow
  processing (denoise, resampling) doesn't split them
- warning and label match the "Jitter buffer" setting

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Some devices (e.g. Samsung) don't report
PROPERTY_SUPPORT_AUDIO_SOURCE_UNPROCESSED but still accept the
unprocessed source. "Unprocessed (forced)" skips the support check
and only falls back to voice recognition if the recorder fails to
initialize.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@teamclouday

Copy link
Copy Markdown
Owner

Thank you for this PR! This playout buffer is exactly what we need. While I'm not knowledgeable enough to review the math of the playout implementation, but I read through the code and it looks good. Also thanks for the other optimizations for latency!

@teamclouday
teamclouday merged commit 8d1bdc2 into teamclouday:main Oct 19, 2026
4 checks passed
@emdaitaj

Copy link
Copy Markdown
Contributor Author

@teamclouday
your welcome
i am c and c++ dev, i dont know rust and kotlin, the whole pr is written by opus 5.5 and tested with my devices... ai code might have bugs but i always use agents to review the code and most bugs are cached .. buy yeah manual review would still be valuable, unfortunately i dont know rust and kotlin and couldn't do it

@wiiznokes

Copy link
Copy Markdown
Contributor

I understand we needed this feature, but i hope not too much AI PR will get merged because the code will became unmaintainable otherwise

ID-VerNe pushed a commit to ID-VerNe/AndroidMic that referenced this pull request Nov 16, 2026
Low, constant latency audio path (playout buffer with drift correction)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants