Repository navigation
The media surface: a dynamic texture channel composited like any widget - #142
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
- media-surface (element 66, widget kind 61) composites producer-pushed RGBA8 textures like any widget; surface (attr 81) binds the model-owned u64 surface id in the runtime-image-id grammar, wired through both engines, the validator, the model contract, LSP docs, and docs previews - Runtime.acquireMediaSurfaceProducer hands out an any-thread handle over a process-lived mailbox (latest-wins staging, generation/owner-tag fencing, no runtime pointers), adopted on the compositor's presented-frame clock with fingerprint damage gates at both the push and adoption boundaries - texture contents stay presentation chrome: adopted textures ride the image pipeline as presentation_only resources the deterministic reference renderer skips (goldens, screenshots, replay marks render the id-derived placeholder), and bit 63 of the ImageId space is reserved for the texture namespace
- pins latest-wins burst adoption on the frame clock, damage short-circuits at both the push and adoption boundaries, viewport clipping plus the surface's own radius mask, and reference-placeholder byte-identity with a producer pushing real frames - pins fingerprint exclusion (a11y line is producer-independent) and record/replay: a session recorded with a live producer replays fingerprint-identical with NO producer attached - pins the thread contract: cross-thread pushes, and a producer outliving its runtime landing in inert process-lived slots with no cross-talk into a successor runtime's claim
- validator teachings pinned: surface is one binding, required, media-surface-scoped; the contract checker requires an integer-producing model binding with the same message - NSUI round-trips the element and attribute under codes 66/81 - the TS markup fixture declares a media-surface bound to a transpiled-core integer field, adopted into the retained tree through the full e2e (record/replay suites included)
- components/media-surface reference page (markup-first, placeholder-policy honest) plus the Media Producers recipe under Mobile & Embedding — the external-renderer/mpv answer's home, explicit about the Zig-tier producer API - native-ui guide and the native-ui skill gain the element row; changelog fragment states the feature, the determinism policy, and the reserved bit-63 id namespace
- textures composite through the GPU packet pipeline; hosts on the software pixel fallback render the reference path's placeholder in this release
…forces - The upload side-channel capped every image at the registered-image slot bound (1 MiB), so a 1920x1080 RGBA8 frame the producer accepted (8 MiB budget) was staged, adopted, and then refused at presentation with InvalidGpuSurfaceImage. - Bounds are now keyed by the id namespace: ordinary registered images keep the avatar-scale registry bound (anything larger is an engine bug the registry already refused), while the reserved media-surface namespace is bounded by the producer's own frame budget, pinned in lockstep with canvas_limits. - New coverage: a real 1080p frame end to end through push, adoption, packet present, and host upload; the exact-budget/one-over boundary refusing at the producer and never downstream; the per-namespace bounds at the PlatformServices seam every host (AppKit included) sits behind.
- The reclaim path overwrote the retained entry without removeGpuSurfaceImage, so hosts that retain copied side-channel textures (AppKit's NSImage store) grew unboundedly as surface ids rotated, and a widget still drawing the reclaimed id could resolve the stale host image instead of its placeholder. - Reclaim now mirrors unregisterCanvasImage's best-effort teardown, issued after the slot lock drops so guarded sections stay bounded memcpys; hosts without the seam and hosts that never uploaded the id are both safe no-ops. - The reclaim-on-exhaustion path also self-deadlocked the moment the entry table first filled: the active-slot scan relocked the non-reentrant spin mutex the adoption loop already held. The scan now reads the held slot lock-free; the other paths (explicit release retains the paused frame deliberately, and a runtime never outlives its platform) need no removal. - New coverage: reclaim removes the host image and the widget serves the placeholder in both engines' resolution paths; a surface-id rotation loop keeps the host store bounded by the channel budget; reclaim is safe on hosts without the upload seam.
- The producer handle type was public only inside media_surface.zig, so the docs' typed mpv callback (producer: media.MediaSurfaceProducer) could not be written as shown; it now rides the runtime root and the SDK root beside Runtime.acquireMediaSurfaceProducer, with the channel budgets exported like the image registry's. - A compile-shaped test drives the docs example's exact signature against the public exports, and a docs-contract step pins the signature in the page and its test mirror in lockstep.
- pushFrame requests ONE coalesced frame through the platform's thread-safe request_frame_fn (the automation watcher's wake path) when it stages new bytes, so an idle demand-driven host adopts a 24/30fps producer instead of stalling and a late-starting producer is adopted promptly; damage-skipped and stale-generation pushes wake nothing, and a per-slot pending flag makes a burst of pushes cost at most one platform call - The binding lives in the slot's process-lived wake half behind its OWN spin mutex with the platform call made UNDER it (the effects executor's abandon-fence doctrine): disarmMediaSurfaceWakes — run-loop exit defer, TestHarness.destroy, embed host destroy — takes the same mutex, so after disarm returns no producer thread is inside the host call and none can start one, preserving the orphan-producer UAF-safety story - frame_requested dispatch now adopts staged frames (adoption's invalidation then arms the prompt gpu-surface frame via the existing noteCanvasImagesChanged path); tests pin the idle wake-adopt round trip, burst coalescing across threads, no-wake on damage-skip/released/invalid pushes, the disarm fence, and teardown with a wake pending
…dlock - The retained-entry reclaim scan locked OTHER mailbox slots while the adoption loop held the drained one: two runtimes with full texture tables adopting concurrently could each hold a different slot and spin on the other's forever (ABBA) — the same class as the shipped self-deadlock fix, one runtime wider - mediaSurfaceHasActiveSlot now snapshots ownership LOCK-FREE: the claim/release sites store the owner-tag/surface-id/active triple with atomic stores (active last, .release) and the scan reads them with atomic loads (active first, .acquire), which is what the memory model honestly supports — per-field tear-free, triple possibly stale, acceptable because it feeds only the reclaim heuristic and self-corrects on the next adoption (documented at the fn) - SpinMutex.lock asserts in Debug/ReleaseSafe that a thread never holds two media-surface mutexes (data or wake) — no nesting means no lock order to violate, so any reintroduction fails loudly in every debug test run instead of hanging a stress run; a bounded two-runtime full-table concurrent-adoption stress test pins the racing shape itself
- emitMediaSurfaceWidget emitted the fit-expanded texture draw with no rectangular clip: cover expands past the widget frame on one axis, the draw-level radius mask is nothing at radius zero, and packet hosts that only mask radii painted the overflow over siblings — the exact overflow emitImageWidget already crops with a push_clip, now mirrored (part id 3, rect = frame) around the media draw - Reference renders are unchanged by construction (the planner crops draws to their requested dst), so goldens stay byte-identical; the rounded case keeps its draw-level mask and gains the same bounds crop - Tests pin the emitted command sequence (placeholder fill, rect clip, cover draw, pop before the sibling paints; no clip for non-cover; radius preserved under the clip) and the reference render's byte containment of the fit-expanded draw
…mbedding 32 MiB - Runtime.media_surface_pixels (4 x 8 MiB inline) becomes one lazy frame-budget allocation per texture entry from the new Options.allocator at first adoption, freed by the new Runtime.deinit (wired through the app runner's five run paths, TestHarness with the leak-checked test allocator, the embed host's destroy, and the docs wasm preview host); zero media use = zero bytes, OOM at adoption drops that frame loudly and retries — measured on the docs wasm preview host as 169.5 MB -> 137.5 MB per component tile (the -32 MiB pool exactly; 12-tile cap: -384 MB), the registered-font-pool regression's twin - Repairs the branch's broken wasm32-freestanding build so the measurement (and the docs preview) compile again: the adoption path's std.debug.print gets the session recorder's freestanding guard (analyzing it dragged std.Io.Threaded in), and the reclaim scan's ownership atomics route through single-threaded-aware accessors (wasm32 lacks 64-bit atomics; with no second thread the plain access is exact) - A FailingAllocator test pins that construction, startup, a live claim, and staged pushes allocate nothing, that a refused allocation degrades without a torn entry, and that the first adoption allocates exactly one frame-budget block reused across frames
…ever chase a mutated options.allocator - Runtime.options is public and mutable, so the lazy media-texture buffer allocating through a live read of options.allocator at adoption and freeing through another at deinit could split the pair across two allocators (silent UB); initAt now captures Options.allocator into a private Runtime.owned_allocator and the adoption alloc, deinit frees, and all ownership docs go through that frozen identity - The zero-allocations-until-adoption test now injects its counting allocator through the real capture site (re-initAt) and flips the frozen allocator's fail_index for the OOM-refusal phase; a new test adopts a frame through a tracked allocator, swaps options.allocator to a poisoned one, and pins deinit's free to the frozen identity with zero activity on the swap-in - Audited the branch's other allocator wiring (app runner run paths, TestHarness, embed host, docs wasm preview host): all pass the allocator at init and only deinit reads it back, so the adoption-vs-deinit pair was the only differing-time read
- Both adoption failure paths (registry full, lazy texture-buffer OOM) cleared slot.staged but kept last_push_fingerprint, so a producer re-pushing byte-identical pixels (a paused video's frame, album art) hit the push dedup gate — no stage, no wake, the frame was lost until the pixels changed; both paths now reset the fingerprint to 0, the same "no previous push" state claim and release use (frameFingerprint maps real hashes away from 0), so an identical retry stages and wakes again - Push-side dedup is untouched: identical pushes between presents still short-circuit at the hash, and the OOM path's comment now states the honest retry contract (the producer's next push retries, not "the next adoption") - Tests drive both drops with a synthetic producer — the OOM path through the frozen owned_allocator's FailingAllocator seam, the registry-full path by manufacturing the raced-release shape the branch guards — then re-push the same bytes and pin that the retry stages, wakes one coalesced frame, and adopts
- The surface and avatar image bindings cast the expression's i64 straight to u64 in both engines, so a signed model field (surface: i64 = -1) trapped safe builds at the @intcast; both seams now refuse negatives with the existing teaching messages (failVoid in the interpreter, the runtimeFail latch in the compiled engine), mirroring the tooltip-delay range guard - Message-payload coercion gets the same guard against its payload's own integer bounds (minInt/maxInt, so signed payloads keep their negative range), and key conversion maps negative integer ids bijectively via @bitcast — keys are identity, so a -1 sentinel is a distinct key, never a trap and never a spurious failure - Negative-binding tests cover surface and avatar through both engines, plus a positive control pinning the u64 id's ride into the widget's image_id
- The committed artifact now bakes in main's CJK font, paste, and pinch work alongside the media surface.
ctate
force-pushed
the
feat/media-surface-channel
branch
from
July 17, 2026 19:01
28bd3d2 to
1232977
Compare
- registerCanvasImageBytes now rejects ids with the reserved media-surface bit before invoking the platform codec, so callers get error.InvalidImageId instead of a codec error and never pay a decode for an unusable id - Test registers PNG bytes under a bit-63 id and asserts the null platform's decode counter stays at zero
- A surface id is any nonzero value below the reserved media-surface texture bit (bit 63, refused at acquire): the producer recipe, the component page, the markup attribute docs plus regenerated vocab JSON, the native-ui skill, and the widget/schema comments now all say so instead of implying any nonzero u64
- Fix the producer recipe's binding example to {player_surface} — markup binding resolution is exact-name, and the model field is player_surface
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Media arc slice 0 — the enabling primitive for video playback, camera preview, and external renderers (the reported mpv request becomes a documented recipe over this API). No codecs in this slice; a synthetic producer proves the machinery.
<media-surface>: a display-only element whose texture is produced outside the widget tree and composited like any widget — layout, clipping, z-order, radius masks.surface="{binding}"binds a model-owned u64 surface id (the effect-key spirit: replay-journal-native, no second name registry); internally the texture lives behind a reserved bit of the flat ImageId space, with both registries loudly rejecting ids that cross the partition (deliberate pre-1.0 contract, changelog-stated).acquireMediaSurfaceProducer(id)on the loop thread;pushFrame/releasefrom any thread. Latest-wins staging (bursts collapse to one adoption on the presented-frame clock), damage gated at both boundaries with producer-side content fingerprints so unchanged frames cost one hash and changed frames never get re-hashed by the planner. RGBA8 first; the format axis is documented for later./media-producerstoolkit-extension recipe including the mpv sketch, vocab/LSP/skill surfaces. Registry: element 66, attr 81, widget-kind 61; fingerprints re-pinned.Synthetic producer battery: latest-wins under bursts, damage short-circuits, clipped compositing with byte-identical reference renders before/after real frames, producer-independent a11y, the no-producer replay pin, cross-thread pushes, and producer-outlives-runtime — each sabotage-verified. Full battery, validate, tooling, examples, ts-core e2e, docs check all green; existing goldens byte-identical.