CI: full pytest suite at 39 of the 45 minute timeout (runtime creep)

[jaylfc]

The test jobs now run 10063 tests in about 39 minutes under xdist (-n auto) against a 45 minute job timeout. Two more minutes of growth and every run starts dying by timeout, which reads as a hang (this already caused one false incident today).

Scope:

1. Profile: run pytest --durations=50 in CI once and attach the output; find the slow tail (suspects: full-app fixtures, real sleeps, subprocess waits).
2. Quick wins: convert real sleeps to fake clocks where trivial, cache the uv venv across runs if not already, check xdist distribution (worksteal scheduler).
3. Structural: if the tail resists, shard tests into 2 parallel jobs (for example by directory hash) so wall time halves; keep the 45 minute timeout.
4. Guardrail: add a soft alert (log line or annotation) when suite wall time exceeds 30 minutes.

Acceptance: suite wall time under 25 minutes per job, or sharded so each job is, and the timeout margin is at least 40 percent.

[jaylfc]

This is now a hard failure, not creep. On PR #2062 (run 29838739851, head e3eaa5c) the job was cancelled:

X The job has exceeded the maximum execution time of 45m0s
test (3.12): .github#1

• test (3.12): 45m15s, CANCELLED
• test (3.13): 35m11s, passed

Same commit, same suite, a 10 minute spread between interpreters, and 3.12 lands the wrong side of the limit. So the failure is nondeterministic: a PR can go red purely on which runner and interpreter it drew, with nothing wrong in the diff. I have re-run the failed job, which will probably pass, and that is exactly the problem. A CI signal that flips on timing teaches everyone to re-run reds instead of reading them, which is how a real failure gets waved through.

Worth noting how this presented: I had a merge watcher on #2062 that correctly reported CI FAILED, and the PR author would reasonably have gone looking for a defect in their own code. There is none.

Options, roughly in order of how much I would prefer them:

1. Find the actual slowdown. 10 minutes between 3.12 and 3.13 on identical code suggests something interpreter-specific rather than uniform growth, which is a lead worth pulling.
2. Split the suite into parallel jobs so wall-clock drops below the limit without hiding the growth.
3. Raise the timeout. Cheapest, and purely deferral: it buys time and guarantees this recurs.

Flagging for prioritisation rather than fixing blind, since the diagnosis in option 1 should come first.

[jaylfc]

Second occurrence, and it disproves the lead I suggested earlier.

On #2090 (a dependabot bump, so effectively unrelated code) the timeout hit test (3.13) at 45m21s while test (3.12) passed at 36m19s. On #2062 earlier today it was the exact reverse: 3.12 timed out at 45m15s while 3.13 passed at 35m11s.

So this is not interpreter-specific, which is what I proposed last time. Same suite, same limit, and which job dies flips between runs. The real picture is simpler and worse:

• The suite takes roughly 36 to 45 minutes depending on runner draw.
• The limit is 45 minutes.
• Whichever job lands on a slower runner crosses the line.

The margin is now smaller than normal runner variance, so any PR can go red at random, regardless of its contents. Two of the four PRs I watched settle today failed this way, both on unrelated changes.

The cost beyond wasted CI minutes: a red result no longer means anything, so the correct response becomes "re-run it", and that habit will eventually wave through a real failure. On #2062 the author would reasonably have hunted for a defect in his own code. There was none.

Revised options, in the order I would take them:

1. Measure where the time goes. 36 minutes of variance-prone runtime is the actual problem; the timeout is only where it becomes visible. A --durations report on one run would show whether this is broad creep or a handful of slow tests.
2. Split the suite across parallel jobs so wall-clock drops well under the limit without hiding the growth.
3. Raise the timeout. Cheapest, buys time, guarantees recurrence, and does nothing about a suite that now takes most of an hour.

I have re-run the failed job on #2090, which will probably pass, which is precisely the problem.