feat(runtime): data-vfx-chain backdrop capture and two-source kernels (luma-matte, noise) by vanceingalls · Pull Request #4330 · heygen-com/hyperframes

vanceingalls · 2026-09-23T11:46:47Z

Follow-up to #4306. Adds the second capture source and the first two-source kernels to the data-vfx-chain runtime.

What

Backdrop capture. A kernel that needs what is behind its host (an adjustment-layer effect) reads a <canvas layoutsubtree class="hf-vfx-src" data-vfx-for="<host id>"> placed as the host's preceding sibling. Resolution order: data-vfx-for sibling → the host's own child wrapper → loud error. self and backdrop share one runtime path; which element fills the texture is the exporter's placement, and VfxEntry.capture reports what was resolved.
Ref second sources (u_src2): a kernel param may name an element id whose subtree carries its own canvas.hf-vfx-src > .hf-vfx-in.
luma-matte (matte ref + mode 1–4) and noise (amount, useColorNoise, clipping; exact-hash port, precision highp int).
Hidden backdrop host passes the captured bitmap through instead of skipping (skipping would delete every layer below the adjustment). A hidden ref/matte captures empty instead of waiting out the 2 s paint ceiling per frame.

Why the deviations from the plan

Measured HTML-in-Canvas rules (children of a layoutsubtree canvas are not painted by the page; only immediate children are capturable; inset:0 captures 0×0): a runtime-injected second staging canvas would hide the htmlInCanvas render-mode hint, so the exporter's wrapper is the staging canvas; a non-immediate .hf-vfx-in child is refused at registration; a 0×0 .hf-vfx-in is reported once.

Verify

bun run --cwd packages/core typecheck:runtime clean
bun run --cwd packages/core test:hyperframe-runtime-ci 59 files / 1266 tests
vfx suites directly: 7 files / 103 tests
Zero-host path unchanged: no new timers, observers, canvases or GL contexts when no data-vfx-chain host exists.

Browser verification (`1a48b48ca`)

9 new puppeteer checks in packages/producer/src/services/vfxDeterminism.test.ts (20/20), Chrome 152 headless with --enable-features=CanvasDrawElement, capability probe guards against a silent pass: backdrop identity reproduces two stacked blocks exactly (PSNR ≥ 40 dB vs a plain-DOM control); a host outside its clip window passes the captured backdrop through; luma-matte modes 1–4 track the matte's alpha/luma within ±4/255 at 4 sample points each; noise at amount 0 equals its input, same seek time is byte-identical, different time differs.

External displacement maps and visible ref sources (`657980307`)

displacement-map gains map as an optional ref param (HfVfxRefParam.optional, a v1.1 amendment). Empty map is the self-referential form every corpus instance uses; a map naming its own host resolves to self. Frag: u_src2 + u_hasSrc2, CPU reference unchanged.
A ref whose layer also paints in AE is marked data-vfx-ref-visible on its wrapper: the runtime uploads the texture AND keeps the bitmap, drawn at the ref's own box. The three capture shapes are one {upload, keepBitmap} mode now.
Tests: jsdom vfx 38 → 44, browser vfxDeterminism 20 → 24 (an external map translates a block by exactly maxH as one run; a self map splits it in two — verified with a negative control). test:hyperframe-runtime-ci 59 files / 1272.
Exporter contract: a visible wrapper's <canvas> must carry CSS width/height equal to the layer box (the runtime rewrites canvas.width/height in device pixels; at DPR 2 an attribute-only canvas doubles its layout footprint).

Known limits

vfx_capture telemetry under-reports backdrop until the producer's markup scan also looks for data-vfx-for. A matte whose box differs from the host's is scaled into the host box (every corpus matte is a full-frame sibling).

🤖 Generated with Claude Code

Interface v1.1's two remaining capture shapes, in the runtime. `backdrop`: a host resolves its source from a sibling `<canvas layoutsubtree class="hf-vfx-src" data-vfx-for="<host id>">` before its own child wrapper, so the same kernel reads either its own pixels or every layer below it — which element the wrapper holds is the exporter's placement, not a def's declaration. `VfxEntry.capture` now reports what was resolved rather than what the defs asked for. The trap that shape creates: a `backdrop` wrapper lives OUTSIDE the host, and children of a `layoutsubtree` canvas are never painted by the page compositor. Skipping a hidden host the way a `self` host is skipped would therefore delete every layer below the adjustment instead of passing it through. A hidden backdrop host now still captures and KEEPS the bitmap, painting no kernel. `ref` params: a node's second source, captured into `u_src2` from the wrapper the exporter emitted around the referenced element. Per node, not per entry, and shared when two nodes name the same element. Every source is drawn at the host's device size so `u_src`/`u_src2`/`.hf-vfx-out` share one coordinate space, and each source's canvas gets its own `paint` wait — paint records are per canvas. A source the clip runtime has hidden captures EMPTY rather than waiting out a paint that cannot come and then throwing "No cached paint record" every frame; an empty matte is also what After Effects shows outside a matte layer's window. Also: a `.hf-vfx-in` that measures 0×0 is now reported once instead of capturing nothing in silence, and one that is not the canvas's immediate child is refused at registration instead of throwing every frame. Both are measured Chrome rules (vault `layoutsubtree-capture-rules`). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

`luma-matte` is the first def with a `ref` param: `out = src · coverage(u_src2)` with Alpha / Alpha Inverted / Luma / Luma Inverted coverage. It is not one After Effects effect but the runtime form of all of them — unsupported track-matte sources, stencils, silhouettes, Set Matte — which the exporter distinguishes by choosing what the host wraps and what `matte` names. `src` is premultiplied, so one scalar multiply covers all four channels, and Luma reads the matte over black, which is what a premultiplied texel already holds. `noise` ports `ADBE Noise` from `AEFX_Noise.metallib`: Bob Jenkins' 1997 integer mix over `(x, y, seed32)`, the two glibc-style LCG rounds, and the `0x7F80`/`0xFF` byte the IR XORs out of them — no table, no state, so it is seek-safe by construction. Reading the metallib's struct metadata rather than the deep dive's prose settles two of its open questions: `ADBE Noise-0002` is `inUseColorNoise` (a checkbox), not a Uniform/Gaussian popup, and the mystery seed floats are `inRandomSeed_High`/`_Low`, the two halves of one 32-bit seed. What the CPU puts in those halves per frame is still unprobed (plan Task 4.3), so the kernel seeds the low half with the frame index; the tests assert distribution and seek determinism, not parity. The shader declares `precision highp int` — the whole port rests on 32-bit wraparound and a fragment shader's integers default to mediump, which ESSL guarantees only 16 bits of. Both defs declare `capture: "self"`, which says a texture is needed and nothing about whose: `noise` on an adjustment layer is the same kernel, and the runtime reads the `backdrop` placement off the DOM. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Task 2.6 Step 5 and Task 2.7's owed browser checks, folded into the existing puppeteer harness: an identity backdrop kernel composites two stacked blocks captured through a data-vfx-for sibling wrapper back over themselves (PSNR >= 40dB vs a plain-DOM control panel); a backdrop host outside its own clip window still keeps those blocks on the wrapper canvas's own bitmap via capturePassThrough; luma-matte's four modes are checked against alpha- and luma-varying matte strips at four sample points each; noise is checked for amount=0 identity, same-time determinism, and a changed pattern when its only seed (seek time) changes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

vanceingalls · 2026-09-23T12:42:55Z

Measured today in After Effects (hyperframes-ae-mcp scripts/diag/probe-noise-seed.mjs, findings §Task 4.3), relevant to the noise kernel's seeding:

The per-frame pattern is a pure function of the layer's local time (comp time minus the layer's start). Shifting a layer's start by 5 frames reproduces the unshifted frames byte-for-byte at matching local time (4 offsets × 65,536 px, zero tolerance). Same absolute comp frame at different local time: chance-level match. So the kernel's t must be layer-local, never the composition frame counter.
Consecutive frames are unrelated fields (r ≈ 0, not a translated copy).
Amount 50 % vs 100 % is uncorrelated, so Amount feeds the hash rather than scaling a fixed field. The exact seed formula stays Tier 2; frames are archived for a brute-force pass against the Jenkins-mix constants if anyone wants to close it.

Two gaps the exporter side found in the `ref` plumbing. 1. `displacement-map` had no `map` ref param, so the exporter's `params.map = "<element id>"` was dropped by `normalizeVfxParams` (it iterates the def) and the node silently self-displaced. `map` is now a ref param, bound to `u_src2` and selected by a new `u_hasSrc2` uniform the runtime sets from whether the ref resolved. Refs gain `optional: true` — empty `map` is the self-referential form every corpus instance uses, while an empty `matte` stays a loud failure — and a ref naming its own host resolves to self rather than capturing the same pixels twice (on a backdrop host it would otherwise have read the layers below as the map). 2. Interface v1.1 assumed a ref source is invisible in After Effects. retro-wave `Logo Anim` layer 5 is layer 4's displacement map AND an enabled, opaque layer, and the runtime cleared its wrapper's bitmap after the upload, deleting it from the frame. A wrapper marked `data-vfx-ref-visible` now keeps its bitmap and is captured at the REF's own box instead of the host's, so it paints at its own size and resolution. The three capture shapes are one `{ upload, keepBitmap }` mode: texture only, bitmap only (a backdrop pass-through), or both. Sampling is unaffected by the box choice, and the browser test proves it rather than assuming it: `drawElementImage` scales the element into the whole canvas, so a 320px-wide matte's four strips still land as four 40px bands in a 160px host. The v1 rule — a ref is scaled into the host's box, not placed in composition space — is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

jrusso1020

Requesting changes for one gap: a visible ref layer disappears from the frame, or freezes, whenever its host isn't painting. Everything else holds up well. The notes below are non-blocking.

What I verified at 65798030

Unit tests pass: vfx.test.ts, lumaMatte.test.ts and noise.test.ts, 68/68.
The browser suite vfxDeterminism.test.ts passes 24/24 on Chrome 152 headless.
These PR-body claims hold in the code:
- Resolution order is the data-vfx-for sibling, then the host's own child wrapper, then a loud error (resolveCaptureSource).
- A non-immediate .hf-vfx-in is refused at registration (:scope > .hf-vfx-in).
- A hidden backdrop host passes its layers through, in both the engine and preview paths.
- The zero-host path adds nothing: module scope only declares lets, and paintVfx returns before arming anything.
No per-frame resource leaks:
- No ImageBitmap is ever created; keepBitmap only skips a clearRect.
- Ref textures are cached per element and released through entrySources in releaseRegistry.
- The paint-wait timer and listener are removed in finish.
The noise hash matches its CPU reference. The shader uses uint throughout, which wraps mod 2^32 in GLSL ES 3.00. I checked refs/noise.ts against a C uint32_t build of the shader's math over 194,480 (a, y, seed) inputs, and every output was identical.
luma-matte modes 1–4: the shader and the CPU reference use the same thresholds and the same Rec.601 weights.

Blocking: a data-vfx-ref-visible layer is only drawn through its host

For a visible ref, the wrapper canvas's bitmap is the only thing that puts that layer on screen, because the children of a layoutsubtree canvas are not painted. But the ref is captured only as one of its host's entrySources:

paintVfx (runtime/vfx.ts ~1187): a hidden non-backdrop host hits continue, so its sources are never captured.
resolveVfxCapture (~966): a hidden backdrop host calls capturePassThrough, which captures entry.src only. sourcesForFrame(entry, true) likewise returns only [entry.src].
An entry with contextLost is skipped outright.

So whenever the ref's AE layer outlives its displacement or matte host's data-start/data-duration window:

Before the host's in-point: the visible layer is missing entirely.
After the host's out-point: the layer shows its last captured frame, because CAPTURE_VISIBLE_SOURCE has keepBitmap: true. With out-of-order or distributed rendering, that frame can come from the wrong time.

Neither case reports anything, which goes against this module's "every failure is loud" stance. A probe with a visible ref and a self host set to visibility:hidden, then paintVfx(0, {engineMode: true}) plus resolve, showed the ref's context received zero draws.

Suggested fix: capture each visible ref on every frame regardless of whether its host is visible: keep the bitmap, and skip the upload when the host isn't painting. Add a test where the host is hidden and the visible ref must still draw.

Non-blocking

display:none refs and mattes (isPaintableHost, applied to src.inner). display isn't inherited, so a .hf-vfx-in inside a display:none ref still computes display:block and counts as paintable. The clip runtime sets display:none for data-hidden and for in-flow timed leaves.
- What goes wrong: in BeginFrame mode the capture waits out the full 2 s ceiling, then reports the fatal and misleading ".hf-vfx-in measures 0×0" error.
- The PR claim: "a hidden ref captures empty" only holds for visibility:hidden.
- Suggested check: inner.checkVisibility({ visibilityProperty: true }), or offsetParent === null.
- I haven't verified whether the exporter emits such mattes.
The noise shader's hash isn't pinned by any test.
- Mutation: I changed the first Jenkins step in noise.frag.ts from >> 13u to >> 14u and rebuilt.
- Result: browser 24/24 and noise.test.ts 13/13 still passed. The browser checks only cover identity at amount 0, same-time determinism, and different-time difference.
- Suggestion: compare a few pixels at amount > 0 against noiseOffsetRef, the way fractal-noise does.
Noise and premultiplied alpha. clamp(out4, 0, 1) doesn't enforce rgb ≤ a, so noise on transparent texels writes invalid premultiplied color into a premultipliedAlpha: true canvas. Separately, clipping=false has no effect, because every target is 8-bit RGBA and clamps on write.
luma-matte with a matte naming its own host works only implicitly (target === host → undefined). u_src2 is then never assigned and falls back to sampler unit 0, which happens to be u_src. Binding it explicitly, or using a u_hasSrc2 like displacement-map does, would make this intentional.
Nit: two doc comments are detached from their functions:
- Around 793-810, the captureSource JSDoc sits above CaptureMode. It also still says keepBitmap is "only for a backdrop", which visible refs made untrue.
- Around 1071-1083, capturePaintedHost's doc sits above awaitSourcePaints.

— Rames

…nits Addresses PR #4330 review feedback (jrusso1020, CHANGES_REQUESTED). Blocking: both the engine-mode and preview-mode capture paths treated "host not paintable" as binary — a backdrop entry got a pass-through capture, everything else was skipped outright, including a ref source that lives on a different, independently-visible element (e.g. retro-wave's Logo Anim layer, which is simultaneously layer 4's displacement map and its own fully opaque visible layer). Adds visibleRefSources()/captureVisibleRefsOnly() and a unified FrameCaptureMode discriminator so a visible ref still gets captured even while its own host is hidden, on both paths. Non-blocking, addressed: - luma-matte's mandatory ref that names its own host now resolves to an explicit RefResolution {kind:"self"} and binds u_src2 to unit 0, instead of relying on an unbound sampler's implicit default. displacement-map's optional self-ref is unaffected (still folds into {kind:"none"}, using its existing u_hasSrc2 shader branch). - noise shader now enforces rgb<=a (valid premultiplied alpha) after adding noise, independent of u_clipping — documented why u_clipping is currently a permanent no-op on this runtime's 8-bit render targets. - added a hash-pinning test for hfJenkinsMix; documented, with a brute-force proof, which one of the reviewer's suggested mutations is unobservable within this fixture's reachable seed range. - fixed two misplaced doc comments. fallow flagged resolveVfxCapture/resolveRefSource as newly over the complexity threshold after this diff; extracted resolveEmptyRefParam() and captureHiddenEntry() to bring both back under. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

jrusso1020

Re-review at b6d51bd9. Almost there: the shape I reported is fixed, but the same bug remains on backdrop hosts.

Prior blocker (a visible ref goes missing or stale while its host is hidden): fixed for non-backdrop hosts, still open on backdrop hosts.

Non-backdrop hosts — fixed. captureHiddenEntry (engine path) and the refOnly mode in frameCaptureMode/sourcesForFrame (preview path) now capture a visible ref while its host is hidden. The two new tests cover both paths.
Backdrop hosts — still broken. Both paths still skip a visible ref when the hidden host is a backdrop:
- captureHiddenEntry returns capturePassThrough(entry) for a backdrop and never reaches captureVisibleRefsOnly.
- frameCaptureMode returns "passThrough" before it considers refs, and sourcesForFrame("passThrough") is [entry.src] only.
- So an adjustment layer whose displacement map is a visible layer still loses that layer outside the adjustment layer's window. It's the same failure the fix targets. A backdrop host with a ref is a supported shape: resolveRefSource has a comment about exactly that case.

I probed it by copying your still captures a visible ref while its own host is hidden test and changing only the host: makeHost(REF_NODE, "adj") plus a data-vfx-for="adj" wrapper as its preceding sibling, host hidden, engine mode, resolved:

PROBE entries 1 errors [] backdropDrawn 1 matteDrawn 0

The pass-through draws, but the visible ref is never drawn and no error is reported. Your non-backdrop test gets matteDrawn 1 on the same setup.

Suggested fix: for a hidden backdrop host, do both captures:

captureHiddenEntry: capturePassThrough(entry) and then captureVisibleRefsOnly(entry), combining the results.
sourcesForFrame("passThrough"): [entry.src, ...visibleRefSources(entry)].
capturePaintedHost's pass-through branch: also call captureVisibleRefsOnly.
Add the probe above as a test, in both modes.

Prior non-blocking items

Luma self-matte: resolved. It is now an explicit {kind:"self"}, u_src2 is bound to unit 0, and there's a test.
Premultiplied alpha: resolved. min(rgb, a) now runs unconditionally, and the permanent u_clipping no-op on 8-bit targets is documented.
Hash pinning: resolved. I mutated c ^= (b >> 5u) to >> 6u and rebuilt, and the new matches refs/noise.ts's hash byte-for-byte test fails. The explanation for why the >>13u first line can't be observed at these seeds holds up.
Detached doc comments: resolved.
isPaintableHost(src.inner) on a ref hidden with display:none on its outer element: still open, non-blocking. display doesn't inherit, so the inner still looks paintable, the preview path waits out the timeout, and then the 0×0 error fires. The new refOnly path now reaches this every frame the host is hidden, not only while it paints. The clip runtime uses visibility: hidden, so it's rare in practice. Checking src.canvas or the ref's owning element would close it.

Tests at this head:

core vfx.test.ts plus src/vfx: 112/112.
producer vfxDeterminism.test.ts in the browser: 25/25.

— Rames

…es through Re-review of #4330 at b6d51bd: the visible-ref-while-hidden fix only covered non-backdrop hosts. captureHiddenEntry returned capturePassThrough(entry) for a backdrop and never reached captureVisibleRefsOnly; frameCaptureMode picked "passThrough" before considering refs and sourcesForFrame("passThrough") was [entry.src] alone. An adjustment layer whose displacement map is itself a visible layer still lost that layer outside the adjustment's window. - captureHiddenEntry runs the pass-through AND the visible-ref capture, OR-ing the results. - sourcesForFrame("passThrough") is the backdrop wrapper plus any visible refs, and capturePaintedHost's pass-through branch also calls captureVisibleRefsOnly. - Tests for the backdrop+visible-ref shape in both engine and preview modes (the reviewer's probe). Also closes the non-blocking display:none item: isPaintableHost(src.inner) missed a ref whose OWNING element is display:none, since display does not inherit — the preview path waited out the 2 s ceiling, then failed with a misleading 0×0 report. New isPaintableSource walks inner's ancestors for display:none (visibility inherits, so one read suffices there); used at both source-side sites. checkVisibility()/offsetParent are absent or always-null in the unit harness's DOM. Test added; verified it fails on the old inner-only check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

jrusso1020

Re-review: approve at 6b67bedf. The one blocker from my previous review (5300141567, b6d51bd9) is resolved.

Blocker: a hidden BACKDROP host with a visible ref skipped the ref. Resolved on both paths.

Engine path: captureHiddenEntry now runs the pass-through and captureVisibleRefsOnly independently rather than returning after the first. Restoring the old early return fails still captures a visible ref while its BACKDROP host is hidden.
Preview path: sourcesForFrame("passThrough") is now [entry.src, ...visibleRefSources(entry)], and capturePaintedHost calls captureVisibleRefsOnly after capturePassThrough. Dropping that call fails the new preview test.

Also fixed: my earlier non-blocking note about display:none on a ref's owning element. isPaintableSource walks the ancestors for display (which doesn't inherit) and reads visibility once at inner (which does). Reverting to isPaintableHost(src.inner) fails treats a matte whose OWNING element is display:none as hidden too.

Tests: core vfx.test.ts plus src/vfx pass 115/115, and the producer vfxDeterminism browser suite passes 25/25.

Non-blocking test gap: one mutation survives. If the pass-through branch of sourcesForFrame goes back to [entry.src] only (so the frame no longer waits on the visible ref's paint), all 115 still pass. The new preview test dispatches paint on both canvases before flushTasks(), so it never checks that the barrier waits for the ref. The test's own comment says it does. To pin it: dispatch only below, flush, assert matteCtx.drawn is empty, then dispatch the matte's paint and assert it drew.

— Rames

vanceingalls and others added 3 commits September 23, 2026 04:44

vanceingalls marked this pull request as ready for review September 23, 2026 12:36

jrusso1020 mentioned this pull request Sep 23, 2026

runtime: startResolver only checks the immediate parent for composition attributes, so a mount wrapped by any element is seeked on the root clock #4334

Open

jrusso1020 requested changes Sep 24, 2026

View reviewed changes

jrusso1020 approved these changes Sep 24, 2026

View reviewed changes

vanceingalls merged commit 84e36fc into main Sep 24, 2026
69 checks passed

vanceingalls deleted the feat/vfx-backdrop branch September 24, 2026 06:25

vanceingalls mentioned this pull request Sep 24, 2026

docs(release): list #4330 in the v0.8.71 release notes #4400

Open

Navigation Menu

feat(runtime): data-vfx-chain backdrop capture and two-source kernels (luma-matte, noise) - #4330

feat(runtime): data-vfx-chain backdrop capture and two-source kernels (luma-matte, noise)#4330
vanceingalls merged 6 commits into
mainfrom
feat/vfx-backdrop

vanceingalls commented Sep 23, 2026 •

edited

Loading

Uh oh!

vanceingalls commented Sep 23, 2026

Uh oh!

jrusso1020 left a comment

Uh oh!

jrusso1020 left a comment

Uh oh!

jrusso1020 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

vanceingalls commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why the deviations from the plan

Verify

Browser verification (1a48b48ca)

External displacement maps and visible ref sources (657980307)

Known limits

Uh oh!

vanceingalls commented Sep 23, 2026

Uh oh!

jrusso1020 left a comment

Choose a reason for hiding this comment

Uh oh!

jrusso1020 left a comment

Choose a reason for hiding this comment

Uh oh!

jrusso1020 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vanceingalls commented Sep 23, 2026 •

edited

Loading

Browser verification (`1a48b48ca`)

External displacement maps and visible ref sources (`657980307`)