Skip to content

perf(producer): load injected video frames by URL during export - #4940

Closed
miguel-heygen wants to merge 4 commits into
mainfrom
perf/render-video-frames-by-url
Closed

miguel-heygen wants to merge 4 commits into
mainfrom
perf/render-video-frames-by-url

Conversation

@miguel-heygen

Copy link
Copy Markdown
Collaborator

What

Exports of compositions with video load each injected video frame by URL from the render's file server,
instead of sending it to the page as a base64 data URI on every captured frame. Output is byte-identical.

Why

During capture, every frame with a visible video clip shipped that clip's frame (a multi-MB JPEG, base64
encoded) inside the CDP evaluate call. The capture page then parsed the string, fetched the data URL and
garbage-collected it on its main thread, frame after frame.

A Chrome trace of the capture browser (single worker, hyperframes-launch from the public launch-films repo,
1080p, BeginFrame capture) over the same 38 s window:

main (835e0c1) this PR
capture-page main thread per frame 88 ms 26 ms
of which handling the CDP message 52 ms 9 ms
frames captured in the window 77 105

Related work

Refs #1630, which turned the URL-served path off because the file server then read files synchronously
and had no Range support, so frame loads starved behind full video downloads and renders timed out. The
file server has since moved to streamed reads with Range support (fileServer.ts), and the extract stage
links extracted frames into the compiled dir, so the resolver #1630 left in place applies again.

How

  • createRenderVideoFrameInjector (render/shared.ts) is the one place the render builds its video frame
    injector, with the compiled-dir frame resolver. The orchestrator calls it; the stale fix(producer): inline base64 frames in injector to unblock video-heavy renders #1630 comment goes.
  • Frames outside the compiled dir still fall back to data URIs, as before. Distributed chunk renders keep
    their own injector for now: their frames live outside the served dir, so routing them needs a plan-layout
    change and comes as a follow-up PR.

Test plan

  • Unit test: the render injector hands the page /__hyperframes_video_frames/..., never a data URI.
    Fails with the resolver removed, passes with it.
  • main's file server answers Range: bytes=100-199 with 206 and the exact bytes, and serves a frame
    through a symlinked frame dir (how the extract stage links frames).
  • Same output: main and this PR render hyperframes-launch to byte-identical MP4s (same md5).
  • Export time, interleaved main/PR runs on the same machine (table below).

TABLE_PLACEHOLDER

Every captured frame of a composition with video sent each visible clip's frame to the page as a
multi-MB base64 data URI inside the CDP evaluate call. The page then parsed it, fetched the data
URL and garbage-collected it on the main thread, frame after frame.

The URL-served path was turned off in #1630 because the file server read files synchronously and
had no Range support, so image loads starved behind full video downloads. The file server now
streams and answers Range requests, and extracted frames are already linked into the compiled
dir, so the injector gets the compiled-dir resolver back. One helper owns the render injector.
…ported

render/shared.ts imports only engine types, and stage tests replace the engine module with a partial
mock. A runtime import of createVideoFrameInjector there broke encodeStage's tests at load time, so
the helper moves into renderOrchestrator.ts, which already imports the engine at runtime.
injectVideoFramesBatch swallowed img.decode() rejections, hid the native video and reported the
frame as painted, so a frame that failed to load became a blank or stale frame in the render. With
frames served by URL a missing file is a 404 in the page rather than a readFile error in Node, so
the batch now rejects with the video id and frame source.
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown

Edit accuracy: accurate 1216 (base branch 1216), smooth 1078 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (1)

Unstable (1)

  • crop-none-px-r0-root-z50: tracking 0.05, pressJump 0, drop 40.07, reload 40.07, render 39.98, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0.13, reload 0.12, render 0.03, renderKey -, undo true, teleport true / tracking 0.05, pressJump 0, drop 0.13, reload 0.12, render 0.03, renderKey -, undo true, teleport true

@miguel-heygen

Copy link
Copy Markdown
Collaborator Author

Closing this after measuring it: loading injected video frames by URL does not make exports measurably faster, so it does not earn its extra moving part.

What I measured (public launch films, main vs this branch, renders interleaved on the same 32-core Linux machine, order flipped each round, 5 rounds per arm, load recorded per run):

film wall median main → PR CPU median main → PR
hyperframes-launch 384 → 301 s (one 863 s main outlier under load) 514 → 514 s
cloud-render-launch 112 → 116 s 314 → 325 s
website-to-hyperframes 128 → 143 s 430 → 436 s
sfx-music-launch 85 → 87 s 223 → 232 s
spacex-launch (no video, control) 147 → 148 s 289 → 273 s
  • CPU per render moves within ±5% on every film, and the no-video control moves just as much, so that is the noise floor.
  • At 1 worker on a quiet machine: hyperframes-launch about 4% faster (390 → 374 s, two pairs), cloud-render-launch unchanged.
  • Peak render memory: no clear change (heap 346 MB vs 329 and 388 MB).
  • The capture page's main thread does less work per frame (Chrome trace: 88 → 26 ms per frame), but the cost moves to fetching and decoding the frame rather than leaving the total.

What stays true and may help whoever picks this up: the file server now streams with Range support, so #1630's reason for inline frames is gone; and a frame that fails to load is swallowed by injectVideoFramesBatch (decode().catch(() => undefined)), which would turn a missing frame into a blank one. The branch stays up with both changes.

Separate finding from these runs: two renders of the same film on main are not always byte-identical, because scene scripts that measure text run before web fonts load. That is being fixed on its own.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant