Skip to content

fix(studio): the timeline viewport gate judges p95 over every measured scroll step - #4847

Merged
miguel-heygen merged 4 commits into
mainfrom
fix/studio-viewport-gate-pooled-p95
Oct 1, 2026
Merged

miguel-heygen merged 4 commits into
mainfrom
fix/studio-viewport-gate-pooled-p95

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

What

The timeline viewport gate now judges scroll responsiveness on the p95 of every measured scroll step pooled, 315 steps over 5 runs at 63 per run, instead of requiring 4 of 5 runs to pass a p95 of 21 steps each. The 75 ms limit, the per-run long-task check and the memory check are unchanged.

Why

The rollback arm (row virtualization off, 1000 rows) failed intermittently on PRs that don't touch the timeline. Evidence pulled from the gate artifacts of 160 main jobs and the failing PR jobs:

  • Not a cold run. The failing runs sit at positions 1 to 5 at random, after the 3 warmups.
  • Not a page regression. I traced main against a build from three days earlier on the same machine, on thread CPU time. Paint, pre-paint and layerize differ by under 5%, and no timeline commit lands near the change.
  • The runner pool changed. Hosted runners fall into two speed classes: a fast one (median step 33 ms) and a slow one (49 ms). The slow class went from 56% of main's jobs (Sep 28 to Sep 30 09:00Z) to 79% (Sep 30 09:00Z to Oct 1 08:00Z). In a sample of main's jobs, its share of runs that fail went from 0 of 135 runs (27 jobs) to 7 of 120 (5.8%, 24 jobs) between those two windows. Over the 80 most recent CI jobs at review time, an independent count found 11 of 320 slow-class runs failing (3.4%); over the full windows the reviewer counted 1 of 255 runs before and 7 of 170 after. The fast class has never failed a run.
  • A run's p95 of 21 steps is its 2nd-worst step. Two slow steps fail a run, so a build whose true p95 is under the limit still fails jobs.

Binomial model of the job failure rate, given the share q of scroll steps over 75 ms:

q (steps over 75 ms) before: 4 of 5 runs, p95 of 21 after: pooled p95 of 315
1% 0.3% 0.0%
2.5% (today's slow runners) 7.6% 0.6%
5% (true p95 at the limit) 43.7% 51.0%
8% (a real breach) 82.4% 98.3%
12% 98.1% 100.0%

Tests

  • The gate and the fade-handle check now share one Chrome launcher (launchStudioChrome in chrome-executable.mjs) instead of two copies of the same block.

  • timeline-viewport-verdict.test.mjs (the verdict now lives in timeline-viewport-verdict.mjs, so it can be tested without Chrome): 16 slow steps of 315 fail and 15 pass; two slow steps in every run pass; one run slow throughout fails; a slow frame interval alone fails; a run short of samples throws instead of reading as fast; the gate fails when any one check fails. Each was run against a broken copy and fails: responsiveness dropped from the exit condition, the sample-count check removed, the verdict judged per run.

  • timelineViewportBudgets.test.ts: the budget owner holds 63 samples per run.

  • Gate sensitivity on a CI runner, with the do-not-merge draft test(studio): inject a 45 ms stall per timeline scroll step to prove the gate fails it #4848: this branch plus a fixed 45 ms main-thread stall on every timeline scroll step.

    • Rollback arm: pooled interaction p95 89.0 ms, FAIL (frame p95 66.7). The old rule fails it too (per-run p95 81 to 100).
    • Virtualized arm with the same stall: pooled p95 58.5 ms, so its 75 ms limit lets a 45 ms per-step stall through (frame p95 goes from 16.7 to 50 ms), as the old rule does too. Whether that limit is the right promise is an open question this PR does not change.
  • This branch on a hosted runner of the slow class (per-run p95 50 to 64 on the rollback arm): pooled p95 60.2 ms on the rollback arm and 33.2 ms on the virtualized arm. Both pass.

  • Added wall time per CI job: the rollback arm goes from 17 to 18 s to 34 s, the virtualized arm from 19 to 22 s to 32 s, and the gate step from 32 to 47 s to 69 s.

Before

The old rule failing the rollback arm on changes that don't touch the timeline (CI gate artifacts):

before

After

The pooled p95 on hosted runners: this branch passes, and the injected stall fails:

after

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Edit accuracy: 557 passing here, 557 on the base branch

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

@miguel-heygen
miguel-heygen force-pushed the fix/studio-viewport-gate-pooled-p95 branch from cbdd8d5 to c48d70d Compare October 1, 2026 10:42
@miguel-heygen
miguel-heygen marked this pull request as ready for review October 1, 2026 12:48
@miguel-heygen
miguel-heygen merged commit c20934a into main Oct 1, 2026
103 of 104 checks passed
@miguel-heygen
miguel-heygen deleted the fix/studio-viewport-gate-pooled-p95 branch October 1, 2026 12:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant