perf(studio): zooming a long timeline re-decodes and redraws less

[miguel-heygen]

What

Zooming the timeline of a long project does less work each time a zoom lands:

• Filmstrips keep their frames across zoom steps. A strip twice as long reuses every frame of the shorter one, so zooming out decodes nothing and zooming in decodes only the new half. While the new width decodes, the strip already on screen stays, instead of dropping to the single poster frame.
• Every filmstrip ends on the clip's last frame, so an end card or a fade-out shows in its strip.
• Waveforms off screen no longer redraw at every zoom. A waveform draws while it is on or near the screen, and when it comes near after a zoom changed it.
• Beat analysis never starts a stage mid-zoom. Each long stage waits until the timeline rests and the browser is idle. Results are unchanged.

Why

Profiling slider zooms on a 60-minute project showed what still ran on the main thread each time a zoom landed: filmstrip frames decoded and scaled again at the new width (the largest cost), every mounted waveform redrawn, and beat analysis of the music landing inside a zoom when it started late (over a second on long audio).

Numbers

Headless Chromium on a Linux box, no GPU, the 60-minute project. Frames per second during each input, main | before this PR (#5109 merged) | this PR, four alternated rounds:

Input	main	before	this PR
Pinch	7.3-12.7	14.3-27.9	15.9-40.8
Slider	2.8-4.6	9.2-14.6	7.8-21.3
Ctrl+wheel	9.9-15.1	12.6-22.6	8.7-24.0
Zoom buttons	12.2-20.1	13.5-18.4	14.0-23.3
Scroll while zoomed	9.4-24.4	10.6-23.7	9.7-25.1

Paired round by round, this PR was faster in 16 of the 20 runs, including all four pinch runs. The machine's load moved between 6 and 20 across rounds, so single runs are noisy. In the per-zoom profile, the zoom's own commit dropped from about 970 ms to about 490 ms over six zooms, and waveform drawing from about 330 ms to about 200 ms (same session, same build flags).

On this project most frames decoded during a zoom are posters for clips that just came into view (159 different clips over six zooms, two decoded twice), so frame reuse pays most on long clips at deep zoom.

How

• thumbnailVideoDecoder.ts samples each slice's left edge (start + duration * i / n) and then the clip's last frame. Left edges are what let strips nest: each step of the power-of-two frame ladder contains every time of the one below it. Decoded frames are shared per source, fit and time, with a user count, and revoked when the last strip holding them is released.
• A shared frame records the time it was actually decoded at (a nearby keyframe can be up to half a slice earlier). A strip reuses it only where it would have accepted that keyframe itself, so frames never run backwards. The last frame is always decoded at its own time, 1 ms before the end, since a time at the very end can decode to nothing.
• A strip whose frames are all shared returns without opening the file.
• The frame ladder now tops out at 32 (a power of two) instead of 33, so its top step shares frames too.
• VideoThumbnail keeps a lease on the strip on screen for as long as it belongs to the clip, and shows it while a new width decodes, so a cache over its budget cannot evict it mid-zoom. The hold is per clip and range: a change in on-screen priority keeps it, and narrowing the clip to one frame lets it go.
• A frame that fails to decode takes the nearest decoded frame before it (the first slot takes the next one), so every slot keeps its time and the last tile still shows the end. The cache is charged only for decoded frames.
• AudioWaveform gates its draw on one shared IntersectionObserver (a screen of margin each side).
• analyzeMusicFromBuffer / analyzeMusicFromUrl take an optional pause awaited before each long stage. Studio passes whenTimelineIdle, which resolves at an idle moment with no zoom moving the timeline.

Known limits

• The tempo detector's stage is a single call (about 1.2 s on long audio), so a zoom that starts inside it still waits for it.
• Shared frames are counted once per strip against the thumbnail cache budget, so cached strips are evicted earlier than their real memory requires (never later).
• A media file replaced at the same URL can mix old and new frames in one strip while another strip of it is still held. Video thumbnails already had no content revision.
• Waveforms and filmstrip strips each have their own notion of "near the screen". The strip tracker reports short strips as fully in view, so it could not gate waveforms without changing how short strips keep their tiles.
• A strip's frames show the left edge of each slice, so a tile can show a frame up to a slice earlier than the time under its centre, and up to about two slices next to a partly visible last tile.
• Ranges shorter than about 1 ms per frame put the end frame at or before the first slice. It is decoded once and shared, never leaked.

Test plan

• Studio and core suites pass.
• New tests for frame reuse, the hold under a cache over its budget, a frame that fails to decode, the poster's and a strip's keyframe windows, revoking shared frames, decoding again when a shared keyframe is too early, the cached-strip fast path, two strips decoding the same frame at once, the end frame, tile mapping, holding the strip during a decode (and not across clips, priority changes or narrowing), the waveform gate, the analysis pause and the idle wait. Each fails with its fix reverted, and passes three runs in a row.
• useMusicBeatAnalysis passing the pause has no hook-level test (no test renders that hook today); the core pause and the idle wait are tested.
• Frame timings with the harness above, alternated in the same session.
• Before/After video of the 60-minute project below.

Before

main (with #5109): a pinch in and out, the slider, Ctrl+wheel and the zoom buttons on the 60-minute project.

before-zoom.mp4

After

This PR, the same zooms.

after-zoom.mp4

[github-actions]

Edit accuracy: accurate 2059 (base branch 2059), smooth 1538 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

[terencecho]

Review at 6910922aa2228e48a9586776abbca2d2aebef22e — requesting changes on the new beat-analysis idle contract.

Recheck idle after the lazy tempo-detector import. packages/core/src/beats/beatDetection.ts:234-237 awaits pause, then awaits loadBpmDetective() before entering its synchronous detect(audioBuffer) stage. On a cold import, a zoom can start while that module is loading; when it resolves, the expensive detector starts mid-zoom without consulting whenTimelineIdle again. I reproduced this deterministically by holding the lazy import, starting markTimelineMotion() after the first idle pause, then releasing the import: detect observed isTimelineMoving() === true. On a long track this is the ~1.2-second stage the PR is intended to keep out of zoom interactions. Please recheck pause after the import and before detect, and pin the delayed-import case in a test. This is an uncovered case in the newly introduced idle guarantee, not a claim that baseline analysis was already idle-aware.

I found no separate issue in the filmstrip frame-sharing/lease cleanup or the waveform visibility gate. Focused beat/waveform/motion tests and the deterministic race repro passed (18/18 in a disposable checkout); the thumbnail audit was read-only because dependencies were not installed there. Required checks passed. The optional Studio viewport gate failed at 69.0/58.3 ms interaction p95; its dense-short fixture contains div clips without media sources, so it does not exercise the changed thumbnail, waveform, or beat paths. A separate open #5151 addresses the wider resting overscan introduced by #5109; I did not count that optional failure as this PR's defect.

— tai

[terencecho]

Review at 2065b15fa91605fb67c4048c09b8b441015db9b1 — requesting changes for a separate filmstrip cache regression.

The delayed-import beat fix is correct: the timeline-idle pause now follows loadBpmDetective() and immediately precedes synchronous detection. The focused beat regression passes (6/6), and focused Studio motion/waveform/thumbnail tests pass (73/73). My earlier suggestion that the filmstrip path was clean was too broad; a base-versus-head reproduction exposed the issue below.

A new timeline session can reuse video frames decoded for an earlier version of the same asset. packages/studio/src/player/lib/thumbnailVideoDecoder.ts:242-250 keys sourceInfos and sharedFrames by source URL, fit and timestamp, then returns an all-shared result without reopening the source. The scheduler's request identity includes sessionEpoch, and its bounded cache retains released results, but the decoder never receives that epoch or an asset revision. On project A → B → A, when A's clip bytes change at the same /api/projects/A/preview/assets/clip.mp4 URL while B is open, the new session can return A's old object URLs and aspect. A deterministic scheduler+decoder probe with the same stable URL and changed dimensions (640×360 → 1280×360) reopened A on the PR base (04736006b1f448c0136fbe147c4bad7e66016b70) and returned the new frame/aspect; at this head it did not reopen A and returned the old frame/aspect. The source URL passed by useRenderClipContent.ts:215-240 is not revisioned in this path, and no production caller invalidates the project cache. Please scope the shared decoder entries to the session/asset revision or explicitly invalidate them on session change, with an A → B → A same-URL test.

All currently listed required checks pass. The optional Comments check fails its comment-share ratchet; that CI signal is not the reason for this changes request.

— tai

[terencecho]

Review at f800a70fc9f6723f1b59d6623e9f894be33803d8 — approved.

Both prior blocking findings are resolved. Beat detection waits for the lazy detector import, then waits for timeline idle immediately before synchronous detection; its focused tests pass 6/6. Video thumbnail decoding now scopes shared frames to the project/session content version and accepts cached source metadata only when that version matches. The same deterministic A → B → A test that failed on 2065b15fa91605fb67c4048c09b8b441015db9b1 passes here: after A's video changes at the same URL, it opens A again and returns the fresh frame/aspect. The new same-version sharing and disposal test passes; focused Studio tests pass 74/74.

Nonblocking follow-up: sourceInfos keeps one latest metadata entry per distinct source URL without a global bound or eviction. This is not a recurrence of the stale-version bug, but consider bounding it for long-lived sessions with many assets. Required Build/Windows checks were still pending at review time; the optional Comments check was red. Neither timing nor that comment-share ratchet is the basis for this code verdict.

— tai

[terencecho]

Review at 341337d5fa5d2a50d68d32066d28467d48db2bad — approved after the comment-only follow-up to my prior substantive review.

The entire delta from f800a70fc9f6723f1b59d6623e9f894be33803d8 removes two comments in packages/core/src/beats/beatDetection.ts; the delayed import, immediate idle pause, thumbnail content-version cache, and tests are byte-identical. No runtime path changed, so the earlier A → B → A cache reproduction and focused test evidence still apply. I did not rerun them for this comment-only head. The removed import/pause comment explained a subtle timing rationale; retaining that context elsewhere would be useful, but its absence is not a code blocker. The current viewport-gate failure is outside this comment delta and does not change the code verdict.

— tai

Verdict: APPROVE
Reasoning: The prior correctness fixes remain unchanged, and this follow-up only removes two comments to satisfy the comment-share check.