Repository navigation
feat(ci,#15709): lake-build heartbeat -- make long builds visible in the step log - #15929
Conversation
…ail pipeline is alive The Lake build step pipelines `lake -R build 2>&1 | tee | tail`, and tail renders its window only at EOF: during the whole build the step log shows NOTHING. The two wedged knot_lean runs (3h13 and 2h10 of total console silence, cache HIT and cache MISS, two runners) were cancelled blind at arbitrary durations -- nobody could tell 90% done from dead. Fix: run the existing pipeline unchanged inside a subshell, and while its pid is alive print elapsed time + the tee'd log's last line every 10 min. `wait` propagates the pipeline's pipefail status, so both invariants are preserved byte-for-byte: the 2026-06-14 pipefail exit-code invariant and the #8915 ::group:: error surfacing. Same fix on all four carriers of the pattern: composite twins (.github/actions/lean-build, lean-axiom) and reusable workflows (.github/workflows/lean-build.yml, lean-axiom.yml). Controls (tick scaled 600s->2s, scratchpad harness): - positive: 8s build -> 3 heartbeats, each showing the live last log line, rc=0 - negative-ok: 1s build -> 0 heartbeats, rc=0 - negative-fail: 1s failing build -> 0 heartbeats, rc=1 propagated through wait, ::group:: fired Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
aucun genre mots-clé fermant dans le body ni les commits ; prev: accepté(s) : #15865 Run vert du garde : ce commentaire bloquant est obsolète. Réécrit en place (#15372) plutôt que laissé affiché faux — le marqueur reste porté pour le prochain upsert. Historique : runs |
|
VERDICT: LGTM (vérifié: lecture intégrale des 4 patches — invariant pipefail tracé à travers le refactor) [Hermes] — lake-build heartbeat #15709 item 6, head Lecture complète des 4 fichiers (2 composite actions + 2 workflows, pattern identique ×4) :
Security scan : 0 match ( |
Path-collision (organ #13359/#13615)Cette PR #15929 (
Le verdict terminal (#15578) signale qu'un cote de la paire est deja sur |
…8189) The #15929 heartbeat slept 600 s between liveness checks, so a lake job held its runner until the next 10-min tick after the build had ended. Measured on the 11 completed Lean jobs of PR #18186: 102 of 140 runner-minutes were that sleep (knot_lean: build done at 07:27:43, job released at 07:37:40). Poll every 5 s and keep the 10-min print cadence. The exit status still comes from `wait` on the pipeline subshell, so the #8915 error group and the 2026-06-14 pipefail invariant are unchanged. Same edit in the three copies: lean-build.yml, lean-axiom.yml and the lean-build composite. Co-authored-by: jsboige <jsboige@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Grain: MED/tooling -- lane myia-po-2026:CoursIA -- prev: DEEP/slides #15865
See #15709 (items 1-5 delivered as the forensic diagnosis comment; this PR delivers item 6, the "éventuelle" instrumentation PR).
What was wrong
The Lake build step pipelines
lake -R build 2>&1 | tee "$RUNNER_TEMP/lake.log" | tail -10, andtailrenders its window only at EOF — during the whole build the step log shows nothing. The two wedged knot_lean runs diagnosed in #15709 (job 103451688746: 3 h 13:54 of total console silence after "Already decompressed 8639 file(s)", cache HIT, runner docker-2; rerun job 103592855941: 2 h 10:03, cache MISS, runner docker-1 — same orphan signature lake+tee+tail+lean in both) were cancelled blind at arbitrary durations: the log file on the runner contained the real progress, but nothing surfaced it, so nobody could tell a slow serialized elaboration from a dead hang. With the CI pool under active famine, a runner pinned hours without a readable signal is exactly the failure mode to make legible.The fix
Run the existing pipeline unchanged inside a subshell, and while the subshell pid is alive, print elapsed time + the tee'd log's last line every 10 min:
[lake-heartbeat] still running after 30 min - last line: ⏹ info: ...— a growing/dated last line distinguishes slow elaboration from a hang in real time, which settles hypothesis 3 of lean(#15698): diagnostiquer le wedge de 3 h 17 de Proof integrity knot_lean #15709 at the next occurrence instead of after the fact.Invariants preserved byte-for-byte (the subshell's exit status is what
waitreturns, pipefail intact):::group::full-log error extraction still fires on failure (verified in control below).Same one-subject fix on all four carriers of the pattern: composite twins
.github/actions/lean-build/action.yml+.github/actions/lean-axiom/action.yml, and reusable workflows.github/workflows/lean-build.yml+.github/workflows/lean-axiom.yml. The lean-knot gate self-covers both action files in itspaths:triggers (lesson #8712), so this PR exercises its own gate.Controls (positive + negative, per the #15709 acceptance)
Harness: exact step structure replicated, tick scaled 600 s → 2 s (scratchpad
hb_control.sh):YAML validity of all four files re-parsed post-edit (
yaml.safe_loadOK ×4).Out of scope (deliberately)
timeout-minutes: 300backstops (fix(ci,#15698): borner les jobs Lean pour convertir un wedge en red check #15706/Aucun des 12 workflows Lean ne borne son job : un wedge de 3 h 17 immobilise la moitie du pool coursia-lean et son check ne conclut jamais #15698): untouched — they did their job.build-jobs: "1"): untouched.Périmètre
4 fichiers, tous sous
.github/:.github/actions/lean-build/action.yml,.github/actions/lean-axiom/action.yml,.github/workflows/lean-build.yml,.github/workflows/lean-axiom.yml. No code, no notebooks, catalogue byte-identical to main.🤖 Generated with Claude Code