perf(ci): run merge-group validation on the hosted lane and measure it (iac#1349 phase 0.5)

[willgriffin]

Phase 0.5 of happyvertical/iac#1349 for this repository — the epic's own stated
precondition before any routing work.

Context

merge_group full mode (Build, Coverage Gate, test-core ×3, test-packages
×3, Typecheck, publish-dry-run prepare) has never run on GitHub-hosted
runners. Everything we know about hosted comes from run
30969467329,
which was pull_request_target / affected mode.

A fresh 12-run measurement of both lanes shows the fleet is the whole wall
clock: merge_group median 47 min (33–71), merge-queue transit per PR median
53 min (mean 83, max 217), and queue wait is 40–80% of fleet job-time on
the slow runs. Observed running concurrency on arc-happyvertical-nodocker
plateaus at 4–6 against 8 nominal slots; in run 31983234153 the fan-out sat
0-running/16-queued for ~10 min after Build finished. Worst single wait: Build
queued 3899 s (65 min) in run 31672411209. Meanwhile hosted is free and
unmetered on this public repo, is a larger box (4 vCPU / 16 GiB vs the pod's
3 CPU / 14 GiB), and did a full clean build in 6.0 min vs a 1005 s median on
the pod.

Scope / Deliverable

• Flip repo variable CI_HOSTED_FALLBACK_ENABLED=true for a bounded window
  covering ≥10 merge_group transits and ≥10 PR runs, then flip it back.
• Record per-job queue (started−created) and duration (completed−started)
  plus run wall clock for both lanes, hosted vs the fleet baseline above, into
  the .github/CI.md Phase-0 table and as a comment here.
• Measure the four matrices with and without max-parallel: 2 — that cap
  exists to be polite to an 8-slot fleet and is pure throttle on hosted (Team
  plan allows 60 concurrent jobs).
• Decide steady state: make hosted the workflow default (delete the
  CI_HOSTED_FALLBACK_ENABLED runs-on expressions outright) or revert.

Expected during the window, not a regression: on pull_request_target the Turbo
shim and the pnpm-store actions/cache are refused (main-scoped write path), so
PR builds are cold — that was 8.1 min in the rehearsal. merge_group and
main get both the shim and the turbo-cache-seed.yml seed.

Acceptance criteria

• Comparison table posted here and in .github/CI.md with run ids, per-job
  queue/duration medians and p90, and run wall clock for ≥10 merge groups and
  ≥10 PR runs on hosted.
• A max-parallel on/off comparison for the four matrix jobs.
• An explicit go/no-go on hosted-as-default. If go, a follow-up issue to make it
  steady state (remove the lever expressions, retune job timeouts per CI.md's
  Job timeouts rule).

Out of scope

• The capacity oracle and the route dispatcher (iac#1349 phases 1–2).
• postgres-tests.yml — its services: container needs the dind lane.
• Raising merge-queue concurrency. chore(policy): reconcile merge-queue authority for SDK migration gate #2262 tracks the max_entries_to_build: 1 /
  check_response_timeout_minutes: 180 drift from the have-config default of
  5/60; do not raise it until this issue shows capacity is elastic — more
  parallel merge groups against a 4–6 concurrency plateau makes transit worse,
  not better.

Links

• Parent epic: happyvertical/iac#1349 (Phase 0 shipped as perf(ci): pin the light validation jobs to GitHub-hosted runners #2236 / PR perf(ci): unblock the validation lane — occupancy, hosted light jobs, fixture hookTimeout #2242)
• chore(policy): reconcile merge-queue authority for SDK migration gate #2262 — merge-queue parameter drift, gated on this
• RFC: decouple merge from release — beta dist-tag channel, affected-scope merge gate, release-grade promotion #2212 — merge ≠ release RFC, deferred pending this
  re-measurement
• docs(ci): record the hosted fallback rehearsal results in CI.md #2227 — the affected-mode hosted rehearsal record in CI.md

[willgriffin]

Measurement window opened.

• CI_HOSTED_FALLBACK_ENABLED=true set at 2026-08-17T03:07:35Z.
• Fleet baseline for comparison is the 12+12-run table in the issue body (runs 31639902580 … 31983234153).
• Re-queued the two in-flight fleet runs so the window starts clean: merge-group entry for fix(core,users): declare the request-scoped database global once in smrt-core #2344 (dequeue → enqueue jump, new run 31990188105, all jobs ubuntu-latest); PR fix(core): emit portable paths in generated smrt-register manifests #2348 validation (31988203465 attempt 2 — note gh run rerun re-evaluates vars, so a cancel + rerun is enough to move a queued run onto hosted; force-cancel was needed because arc-queued jobs never acknowledged the cancel).
• Confirmed no smrt job remains queued on arc-happyvertical-nodocker.

Will post the hosted per-job queue/duration table here after ≥10 merge_group transits and ≥10 PR runs.

[willgriffin]

Window 1 result: 03:07:35Z → 04:36:25Z (89 min), lever reverted on a blocker

Merge group, full mode, hosted — first ever measurement (run 31990188105, PR #2344, green):

	fleet, 12-run median	hosted
wall clock	47 min (33–71)	18.8 min
Build	1033 s	392 s
Publish Dry Run prepare	1005 s	410 s
Coverage Gate	442 hit / 1391 p90	726 (miss)
test-core shards	344–410 s	185–210 s
test-packages shards	134–204 hit / 493–844 p90	226–360 s
Typecheck	140 hit / 803 p90	269 s
queue wait, downstream	p90 900–2100 s	2–7 s; 190–231 s only on shard 3/3 of each matrix, i.e. max-parallel: 2

#2344 merged 19 min after enqueue (fleet median transit 53 min).

PR (pull_request_target, affected mode) hosted runs: 12–35 min wall with 2–7 s queue. The wall clock is now the lane's shape, not capacity: affected-package-tests needs affected-core-tests, and both matrices are max-parallel: 2, so 3 shards run in two rounds twice; longest single job was 9–10 min. Removing the chaining and the cap on hosted should give ~10–12 min PR validation. Second merge group (#2347, 31993458544) was 15 min to failure — same shape.

Blocker (why the lever came back off): #2352 — the two OIDC "shared DuckDB root handle" tests (profiles + users) fail on hosted in ~50–70% of runs (3 of 4 PR runs, 1 of 2 merge groups) as a prepared-statement error or a silent native worker abort; they pass on the fleet. PR #2353 fixes the pre-lock statement race but the abort persists on its own hosted run. Hosted-specific in the sense that 4-vCPU hosted timing hits it reliably — not a hosted incompatibility.

Verdict so far: hosted is ~2.5× faster per merge group and ~zero queue; go/no-go on steady state waits on #2352 plus the ≥10/≥10 sample. Next window: after #2352 lands (or with that shard quarantined), and remove max-parallel: 2 + the core→package chain in the same window to measure the real floor.

[willgriffin]

Window 2 opened at the timestamp on the variable above (CI_HOSTED_FALLBACK_ENABLED=true), now that #2353 has landed (merged 10:28Z; #2352 closed). Nothing was in flight on the fleet at flip time, so no re-queues needed.

Two more fleet datapoints from between windows, same shapes as the hosted ones in window 1: merge_group #2347 62 min (Build 1240 s, max queue 1088 s), #2348 33 min, #2353 55 min (max queue 1298 s) — vs 18.8 min hosted.

Counting from here toward the ≥10 merge_group / ≥10 PR table. max-parallel: 2 and the core→package needs chain are still in place, so PR runs will keep reading ~30 min for ~10 min of work until that change is made; I'll measure this window as-is and propose the shape change as its own PR.

[willgriffin]

Shape change (#2383 / PR #2386) landed 18:02Z and executed for the first time in its own merge group, run 32052006473: all six test-core / test-packages shards started at the same second with 3 s queue (shard 3/3 was 190–231 s before), merge_group wall clock 12.8 min (was 18.8 min hosted in window 1; fleet median 47). #2387's merge group earlier today: 17 min hosted, still under the old cap.

Window 2 continues with the lever on; PR-side numbers under the new shape will accumulate from here.