DeviceLink WSS remains the authorized signaling/control transport. WebRTC media and its input channel use ICE, with the existing JPEG path retained when video cannot connect. File transfer reuses the ICE configuration through an independent data-only connection; object storage remains its fallback (see below).
Each viewer attempt requests GET /api/device-link/ice-servers from its authenticated
device-link service. The controlled Desktop independently requests that same API
while preparing its capture window. API credentials stay in Main / native Mobile;
only short-term TURN allocation credentials cross the dedicated capture/WebView
bridge. These scoped WebRTC credentials cannot authorize desktop input or invoke
business APIs. They are never embedded in static HTML, persisted or logged.
Response: {iceServers: [{urls: string[], username: string, credential: string}], expiresAt: string | null}. At most four entries and four STUN/TURN URLs per entry;
URL scheme/length/ports and credential lifetime are checked before bridge delivery.
Require two minutes of remaining lifetime for cold capture/SDP/ICE headroom.
The owning backend supplies its deployment's node pool; no new user region setting.
Missing endpoint, empty configuration, invalid response and an eight-second timeout fall back to the old public STUN list. With a valid self-hosted list, public STUN is not added. All configured nodes participate in ICE checks; a dead first node does not prevent using candidates from the others. URI ordering is not a promise of priority. Candidate statistics already distinguish direct, relay and JPEG paths.
Every existing bounded media retry fetches fresh configuration on both peers.
No configuration cache, WSS reconnect or new global retry loop. All Desktop/Mobile
video and file callers share the configuration budget in
remoteDesktopIce.ts.
The WebView and iOS native receiver allow another 500 ms for bridge delivery
(8.5 seconds total). Lease/generation/attempt checks reject stale
responses; explicit stop takes precedence. A failed media attempt affects only its
owner. Credential expiry may require rebuilding the connection; this change does
not promise uninterrupted in-place renewal of TURN allocations.
Regression coverage includes slow configuration, bounded fallback and late-response isolation in the resolver tests and viewer tests.
Old clients continue unchanged. New viewer + old Desktop can still use viewer-side TURN; old viewer + new Desktop can use host-side TURN. Existing full-SDP and trickle ICE capabilities stay unchanged. The desktop-video connectivity change adds no new wire kinds or native dependencies. The independent file transport below has its own authorized IPC channel.
New iOS binaries expose CindyRemotePresentation.nativeVideo. They receive and
decode WebRTC in RemoteDesktopReceiver, and display sample buffers through
RemoteDesktopVideoView / AVKit. The WebView retains input, cursor and viewport
geometry only. Older binaries and Android retain the browser receiver; JPEG
compatibility frames remain available on every platform. The native receiver uses
the same full-SDP/trickle-ICE signaling adapter, scoped TURN configuration and
bounded retry policy. No protocol version or service endpoint changes.
While system PiP is actually active and host presentation authorization is confirmed, the native DataChannel answers the existing viewing challenges without React or WebView JavaScript. Desktop preserves only an authorized, non-controlling presentation across transient signaling loss; ordinary subscriptions and file transfers still stop. The existing 12-second lease expires if media pongs stop. Explicit disconnect, revocation, disabling remote control and host shutdown still stop the presentation immediately. An offline event for a different peer cannot terminate the active viewer. The native receiver retains the host's single pending challenge if it arrives before PiP entry, and answers only after both AVKit active presentation and host authorization are confirmed. Voice-input cleanup releases only its own recording session; background entry and delayed prewarm cleanup must not deactivate PiP's playback session.
Run in background is an explicit phone-local preference (default off), separate
from the current AVKit presentation state and the user's control choice. Toggling
it does not enter PiP or release control, and audio renegotiation does not disable
the preference. Only actual entry uses the host's view-only presentation handoff;
returning to an active, visible fullscreen viewer restores the previous control
choice through normal host authorization. A cancelled Home gesture also restores
that choice after any pending preparation settles. Restoring fullscreen does not clear
the preference. While enabled, leaving the desktop route minimizes into PiP;
the foreground viewer arms AVKit after its first frame. Home entry is owned by
AVKit automatic entry: do not manually start from willResignActive, because the
source UIScene may already be inactive even while UIApplication reports active.
Explicit in-app entry requires the source scene to be foreground-active. The host
handoff runs in parallel, without delaying AVKit for its network reply.
The sample-buffer delegate reports no playable content until a frame is rendered.
First-frame and teardown transitions invalidate AVKit's cached playback state;
returning inline refreshes that state after the PiP stop completes. A cached old
frame alone does not make a reconnecting receiver ready for PiP.
Keep the AVKit sample-buffer projection at the native viewport's bounds, separate
from the inline layer's fitted/panned/zoomed rectangle. Both renderers share one
decoded sample buffer; their readiness must not block each other. When the viewer
is zoomed in, only the projection's frame content follows the zoom: it receives
the desktop region visible inline, copied from the decoded NV12 frame
(RemoteDesktopPresentationCrop). The projection layer's geometry stays fixed, so
PiP keeps the browsing zoom without moving the source; a fitted, unzoomed view
passes the frame through unchanged. Only the system
projection receives frames while inline is hidden or the app is inactive. Restore
replays the latest frame into the inline renderer before completing the visible
source handoff. In physical-device A/B testing, a fitted or transformed source
missed interactive Home entry even while AVKit reported PiP possible; a fixed
viewport-sized source started successfully. This is an observed regression guard,
not a claim about Apple's private eligibility rules. Test actual upward Home
gestures after restore: an automation Home-button event does not cover that path.
System readiness alone never authorizes viewing challenge replies.
Both JS and native background cleanup bound an unfinished handoff to four seconds.
The signaling connection's ordinary 2.5-second background grace must wait for an
in-flight presentation handoff, capped at four seconds after background entry.
Heavy subscriptions are still released immediately. Success, failure, cancellation
and the handoff deadline release this wait; it never authorizes media or renews a lease.
Rapid Home re-entry waits for pending fullscreen control restoration before sending
a new presentation request; expired lease/generation results cannot revive it.
Fullscreen restoration waits for signaling to be online before relinquishing its
acknowledged background-viewing authorization or requesting input control. While
reconnecting, the native stream keeps that authorization; another Home entry can
reuse it. A stopped signaling socket alone must not destroy this authorized stream.
One account-generation-scoped root host retains the same media surface across
routes. A minimized desktop is removed from navigation history without changing
the other pages or their parameters. Fullscreen restoration opens one desktop
entry above the current page; Back returns to that page. Delayed PiP callbacks
are bound to their source route and cannot pop a newly opened page. The retained
surface only becomes visible once its desktop route is current. AVKit restore
completion waits for JS route readiness and a visible, laid-out native source
(including ancestor opacity), not merely attachment to a window. A generation-scoped
check covers parent-only visibility commits and expires at the existing five-second
restore deadline.
Closing a detached PiP window or explicitly disconnecting ends the connection,
but keeps the preference. Back-to-PiP does not run lock-on-exit; disconnect does.
Preparation has a bounded deadline and never renews a controlling lease.
Closing PiP in the background stops native media even if JS is suspended. Network changes that require a new SDP exchange recover after foreground signaling returns; continuous background renegotiation is not promised. Frames and ICE credentials stay in memory, with no recording or new media store.
The WebRTC SDK and Swift module change require a rebuilt iOS binary (cold update), not OTA alone. Verify silent and audible streams, explicit stop/revoke, switching apps for several minutes, PiP close/restore, network loss, zoom/keyboard geometry, and Light/Dark on a physical phone. Unit tests and simulator builds do not establish background PiP acceptance. Roll out the Desktop signaling-loss fix with the native phone build; older Desktop hosts may still stop media when signaling disconnects.
Run node apps/mobile/scripts/test-remote-desktop-presentation-crop.mjs on macOS
to check the PiP zoom region geometry, 4:2:0 alignment and NV12 row copies.
Run node apps/mobile/scripts/test-remote-desktop-receiver.mjs on macOS to
compile and execute the production receiver against test-only UIKit/WebRTC doubles.
It covers attempt isolation, late SDP/ICE/frame callbacks, idempotent stop,
disconnect timer cancellation and presentation challenge authorization. The doubles
control SDK callbacks; they do not validate framework ABI, decoding or AVKit behavior.
iOS 27 SDK builds also require the scene lifecycle. Expo 57.0.23 or newer and
expo-build-properties's ios.enableSceneSupport generate Expo's scene delegate
manifest and move window startup out of the legacy app delegate. Verify actual
device launch: compilation alone does not detect UIKit's missing-scene launch trap.
Unit tests cover invalid/expired tickets, old/disabled backends, bounded requests,
fresh credentials/backup lists on retries, and stop/late-response isolation.
The Server repository contains the coturn deployment template and matching API
contract (docs/device-link-server.md, infra/turn/README.md).
For a real relay test, store an authenticated API response in a task-specific temporary directory outside this repository (mode 0600), then run:
node scripts/remote-desktop-turn-smoke.mjs /absolute/external/ice-config.json /absolute/path/to/chromeThe probe uses an isolated sandboxed browser, forces relay on both peers, checks 16 KiB data round-trip and the selected relay pair for each TURN URL, then repeats with an unreachable first node. Output excludes URLs, IPs, credentials and SDP. Delete the temporary credential file after use. This operator probe makes real network requests and is intentionally excluded from unit tests.
Local test evidence for this change: coturn 4.18.0, two loopback nodes, real Chrome, UDP/TCP for both nodes and unreachable-primary case passed. Separately, coturn's client verified a temporary test CA and received 40 TLS-relayed messages using the deployment template with loopback-only test overrides. This proves TURN data flow with the Server credential algorithm; it does not prove platform UI behavior, publicly trusted browser TLS, the Linux container deployment, or Mainland China reachability. For rollout, repeat on the deployed nodes and real Desktop/Mobile with trusted TLS, node failure, >1-hour sessions, two viewers and network switching. Compare STUN-only versus configured TURN on identical network pairs, recording sample count, connection success, first moving frame time, selected path, RTT and recovery time.
Before external QA, deploy the matching ICE API to the test environment and inject its TURN node configuration through deployment secrets. Distribute Desktop and Mobile builds from this change that use that environment. Both devices must use the same account/session realm. A loopback proxy or an inspector-only endpoint override on a developer machine is not a usable QA environment. Keep login realm discovery intact; Desktop local-server mode pins both realms to the local manifest and is unsuitable for testing production enterprise SSO across realms.
On 2026-09-09, a macOS Desktop and an iOS simulator on the same machine displayed
real desktop video and delivered a pointer movement to the expected OS position.
With a separately hosted public coturn node, forced UDP relay and TCP relay with
an unreachable first candidate server connected. Closing an established media
connection recovered via UDP relay in approximately 16 seconds. The deployed ACL
was corrected to omit denied-peer-ip=::, which coturn interprets as an unbounded
range; coturn already rejects unspecified peers before its ACL checks.
These are functional probes, not a network-quality acceptance result. A direct sample had 1 ms RTT and a UDP-relay sample had 84 ms RTT; neither is an average or end-to-end input latency. Receiver loss and jitter were not collected. Publicly trusted TLS, Linux Compose, physical phones, Windows/Android, concurrent viewers, active-node failover, and hour-long sessions still need QA coverage.
For each network pair, record client versions, session realm, transport, attempts, successes, first moving frame time, RTT p50/p95, receiver packet-loss deltas, jitter, frame rate/freezes and recovery duration. Distinguish connection recovery from initial selection of a reachable backup, and RTT from input-to-visible-response latency. Test the previous STUN-only behavior on the same pairs. Include JPEG and object-storage transfer regression checks; do not report unavailable metrics as zero.
device-link:file-peer is an allowlisted DeviceLink RPC, with a 30-second request
budget and versioned caps, offer, open, close actions. Signaling keeps the
existing account, enabled-host and controller-revocation checks. It introduces no
relay envelope kind or server file API. Both peers use the existing ICE endpoint;
ICE chooses a direct or TURN path. The file channel does not share desktop video,
input handling, capture permissions or capture-process lifetime.
Desktop prepares the file-transfer host (up to 10 seconds) and fetches ICE configuration (up to 8 seconds) concurrently, then rechecks the owning connection before issuing the 15-second media command. This keeps the cold-start stages within 25 seconds and retains headroom inside the unchanged 30-second RPC budget.
File access has two layers. packages/device-link/src/fileAccess.ts owns the
shared directory/text operation facade and whole-file transfer selection; Desktop
and Mobile adapters provide platform I/O. Sidebar downloads, message files/media,
Mobile export/share and requested HTML resources use this policy. Directory listing
keeps the existing remote-op and separates complete enumeration from display
filtering. Bounded text previews retain binary detection, truncation and gzip;
they are preview projections rather than whole-file downloads.
The first directory/text request uses the existing WSS path immediately and starts
one background file-peer setup. Ready peers carry subsequent reads on a separate
bounded data channel; transport failure falls back to WSS, while a host error remains
a host error. The same authorized dispatch runs on the host for either transport.
Whole-file reads request prepareOnly after host authorization. Up to 64 KiB,
including empty files, returns inline bytes; larger files attempt the reusable
file WebRTC connection, then OSS. The peer limit is 2 GiB; range-streaming
media intentionally use OSS, selected before downloading any peer bytes. Workdir
downloads probe caps.fileRead first: old hosts retain two-phase export jobs, so
large uploads do not regress to a single relay invocation. fileUrl resolves a
workdir-relative reference on the host, never guesses nested SSH as local storage.
Nested SSH whole-file export remains unsupported as before; media URLs with an
explicit verified SSH session retain their existing SSH materialization path.
Mobile uses a separate trusted WebView with a
build-time-generated script (node scripts/file-peer-runtime.mjs), never native
Function.toString(). User HTML receives no native file or transport bridge.
Missing peer capability, connection or transfer failure falls back to OSS; explicit
cancellation/account change cancels the operation instead. Old hosts remain usable
through OSS, but the direct file path requires an updated host.
Transport errors do not bypass host authorization. Cancelled/obsolete reads do not start fallback or deliver completed results; disposable peer files are released. Legacy OSS export jobs and shared cache fills may finish in the background after a consumer leaves. They do not tear down another consumer's transfer or the relay. Directory/text/index caches and file-download identities include account/device scope; unscoped Mobile v1 directory/snippet caches are not read. Preview snapshots still own their copied files independently from the short-lived transfer staging.
An open request resolves the same authorized media URL as OSS. Only Main resolves
paths, checks the effective size limit and opens the descriptor. The renderer sees
an opaque one-use ticket. Files are limited to 2 GiB, transferred in 16 KiB blocks
with at most 64 outstanding blocks on negotiated files-v2 (1 MiB), and checked for exact size, offsets and source
stat changes through EOF. This is not a persistent content-hash cache. Changed files
fail and follow the existing fallback behavior. Each source and sink rechecks its
connection owner; revoking one controller closes only that controller's transfers.
Reads queue per Desktop peer and per Mobile file connection; a busy channel alone does
not cause an OSS fallback. Queued cancellation skips that read without interrupting its
predecessor. Connections are bounded, reusable for sequential files and expire after 60 seconds
idle. Mobile staging has a 4 GiB aggregate cap and a five-minute lifetime; consumers
copy into their existing cache/preview ownership. Backgrounding cancels in-flight
work, but completed files retain their normal lifetime. Account changes remove them.
Local Desktop HTML opens directly with file:// in the sidebar or system browser,
following the existing opening preference. It does not enumerate or copy its parent
directory. Remote HTML uses a viewer-local loopback HTTP server. Each browser request
stats/reads only the requested resource through existing file access; no directory
listing, total file-count or aggregate directory-size check precedes opening.
Resources are transferred as whole files before responding, not HTTP range streams.
The preview layer has no separate 100 MiB resource cap; missing or failed resources fail
individually while the page remains usable. Reloads may observe changed source files;
there is no atomic directory snapshot. Relative resources stay within the entry's
parent directory; hidden paths are excluded and realpath checks prevent symlink
escape outside that root.
This does not execute a remote site's backend.
Desktop retains at most eight active previews; a new open evicts the oldest when
full. External tab closure is not observable. An evicted page must be reopened;
the two-hour expiry and concurrent preparation bound still apply. Each HTTP response
owns its staging files, cleaned after response completion. Shared cache fills may
complete after a preview closes without cancelling other consumers.
Desktop and Mobile share the snapshot CSP and parser-first device API guard in
maker-shared/file-preview. UTF-8 .html/.htm pages receive the guard; Desktop also
serves the CSP as a response header. Other encodings and XML documents (XHTML/SVG) preserve their MIME
and bytes, receive the response policy, and do not receive an HTML script prolog.
Same-origin assets and requests remain usable,
while external subresources, fetch and forms are blocked. The
snapshot policy also explicitly denies workers, preventing Service Worker registration
from surviving a temporary loopback origin. This is not a zero-egress
sandbox: documents without the prolog and the shared guard's residual child realms can access WebRTC, and an
external browser's top-level navigation is outside the loopback server's control.
Mobile uses native request/response callbacks for on-demand resources. The legacy
start(files) API remains available, and JS on older native binaries retains the
previous directory-snapshot behavior and limits. On-demand loading requires a new
iOS/Android native build; the WebRTC transport itself reuses the existing WebView dependency.
Validation entrypoints: node scripts/file-peer-smoke.mjs <Chrome executable>
checks the shipped WebView script over a real local WebRTC connection, including
empty/block-boundary files, sequential reuse and sink failures. Protocol, Main
permission/descriptor and Mobile staging tests cover bounded failure paths. This
local browser probe does not establish deployed TURN, physical phone, network
switching or release-build acceptance; test those separately with matching accounts
and record versions and the actual selected path.
Protocol version 1 remains unchanged. Optional caps.streaming selects files-v2
and its separate reads-v1 request channel; an old host uses files-v1 with the
original 16-block batches. V2 replenishes credit after every 32 completed sink
writes, overlapping network reads and disk writes without unbounded buffering.
The request channel caps each payload at 4 MiB and outstanding requests at four.
It accepts directory/text reads, clipboard read/write chunks, and negotiated
attachment staging only. Clipboard begin/copy/commit and ordinary mutations stay
on WSS. Identical clipboard staging writes can be replayed after a lost response;
commit remains single-use, so fallback cannot paste twice.
Failed peer attempts enter per-device cooldown: 30 seconds initially, exponential
backoff capped at five minutes. Queued transfers provide a single retry after the
cooldown; success resets the backoff. Cancellation/account changes reject instead
of starting OSS. Cooldown has no background polling and does not stop another
device's connection. Local diagnostics record selected direct/relay path, transport
protocol, RTT, setup time, bytes, transfer time, throughput and fallback stage;
candidate addresses, paths and payloads are not added to these metrics.
The ICE configuration diagnostic counts TURN URLs by client transport (UDP, TCP, TLS)
without hosts or credentials.
Transfers that outlast one second also sample the runtime stats once per second on
both ends (Desktop main log device-link:filePeer, Mobile opt-in Debug log): candidate
kinds and relay protocol, gathered candidate / relay-transport / pair-state counts,
selected-pair/transport byte and packet counters, bitrate
estimates, data-channel bufferedAmount, and application progress — receiver
arrived/written bytes, bridge write latency and idle time; sender reads and time spent
waiting for credit. After EOF the sender keeps sampling until the channel buffer drains
(at most 30 seconds). Comparing the two ends separates slow network delivery, relay
loss and receiver-side write stalls. Sampling uses a separate probe that neither renews
the idle deadline nor closes the connection when it times out
(filePeer.ts, tests in
filePeer.test.ts).
A probe that times out only skips its sample — monitoring continues while the transfer
runs — and transfer chunks never refresh its five-second budget. Even without samples
the drain phase stays bounded at 30 seconds, and a new transfer on the connection
supersedes the previous file's drain accounting because bufferedAmount covers the
whole channel.
Engines expose different stats subsets; missing metrics are omitted, not reported as 0.
Optional caps.attachments enables controller-to-host byte staging for Desktop and
Mobile, complementing the existing host-to-controller download path. Staging failure
abandons the upload and falls back to OSS before the message is sent. Shared-task
guests continue using their existing OSS scope. New peer references are sent only
to a capable host. The receiving Main process checks size and SHA-256 before returning
the reference, then normal message acceptance materializes it through the existing
media/file ownership logic. Message acceptance itself remains on WSS.
Hosts that also advertise caps.streamAttachments accept up to three 1 MiB blocks in
flight, sent as raw RPC bodies instead of base64 fields; other hosts keep the
one-block-at-a-time base64 upload (protocol-compatibility.md).
The host inbox lives under the current owner's userData namespace. Tickets bind to the source controller and are rechecked when materialized. Completed staging survives host restarts for seven days; incomplete staging expires after an hour. Admission sweeps expired entries and bounds retained reservations to 4 GiB / 128 uploads, with space for the consumer copy plus 256 MiB. This inbox is transport staging, not a new media store. Mobile retains local source bytes in its durable outbox for retry.
Reproducible local probe: node scripts/file-peer-smoke.mjs <Chrome> --benchmark
adds a 1 ms timer to each simulated source/sink bridge operation. On 2026-09-26,
the 8 MiB case took 4433 ms with V1 and 2834 ms with V2 (about 36% less elapsed
time). Local ICE setup was 203 / 124 ms respectively; these figures exclude cloud
signaling, TURN config fetch and app cold start and are not a WAN speed claim.
The --large probe also passed 101 MiB, exact bytes, EOF, sink failure, fragmented
Unicode RPC, four concurrent RPCs and RPC during file transfer. Actual Mobile
WebView/native I/O, deployed TURN and network switching still require device QA.
Large-file transport retains protocol version 1 and advertises optional caps.maxBytes.
Old peers keep their previous limits and rejected peer reads retain the existing OSS fallback.
Both receivers reserve disk space for the incoming file and its consumer copy plus 256 MiB
headroom; preview copies and Mobile OSS downloads also check disk headroom. WebRTC receive
commands use a 60-second deadline renewed by successful disk writes rather than a ten-minute
total deadline. Mobile on-demand HTTP requests allow up to two hours including queueing,
transfer and response consumption; cancellation/backgrounding still closes them immediately.
This does not add Range streaming or make very large HTML document parsing memory-bounded.
Run pnpm test:peer-transfer from an installed checkout (Node 22.12+, pnpm).
No login, second device or manual network toggling is needed. The runner discovers
Chrome (or installs Playwright Chromium), runs shared/Desktop/Mobile business
tests and type checks, checks the generated WebView source, then exercises the
production browser runtime with V1/V2, 101 MiB files, Unicode RPC and a slow bridge.
It installs pinned node-turn@0.0.6 into an isolated temporary directory, starts
a UDP TURN server bound only to loopback with random in-memory credentials, and
forces relay-only ICE. Selected-pair stats must report relay; direct connectivity
cannot silently make the relay test pass. This fixture is not a shipping dependency.
The fault probe disconnects a real peer during a random 2 MiB transfer, validates the HTTP fallback's SHA-256 through the production file-read policy, confirms repeated reads skip peer during cooldown, advances an injected cooldown clock, and verifies recovery, slow writes, cancellation, reverse attachment staging and connection isolation. HTTP storage and attachment receiver are test adapters; real Main attachment persistence/ownership and clipboard replay behavior are covered separately by the business tests. These are layered integration tests, not a full logged-in app UI test.
Each stage has a deadline; failures exit nonzero. Children and fixture directories
are cleaned up automatically. A unique OS temporary directory retains stage logs,
report.json (including browser measurements) and report.md; its path is printed.
The report explicitly excludes deployed TURN, TURN TCP/TLS, WAN NAT, physical
network switching, real OSS and native iOS/Android WebView. A local pass must not
be presented as passing those environments. No user files or account credentials
are read by this runner.