Repository navigation
ci(tests): run test suites in 10 parallel lanes - #1238
Merged
Merged
Conversation
The 83 suites ran as serial steps (median 49s on Blacksmith). They now live in scripts/tests/ci-lanes.txt and run as 10 balanced lanes in one GitHub parallel group. Each lane runs every suite (one log group each), then fails and names the suites that failed. A coverage check fails when a scripts/tests/test_* file is in no lane. .github/scripts/ci-tests-lanes.py rebalances lanes from a run's timings. Experiment aeon#1237: 21-22s on Blacksmith.
…everted next)" This reverts commit c4d4b14.
Lanes rebalanced from the first parallel run's suite-time lines (each lane about 11s of suite time). Each suite's stderr now goes into stdout, so its lines stay inside that suite's log group instead of drifting into another.
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
scripts/tests/ci-lanes.txt(<lane> | <name> | <command>) and run as 10 lanes in one GitHubparallel:group (max 10 at once; 10 lanes, not 83 parallel steps, because Blacksmith adds ~0.2s teardown per parallel step).scripts/tests/run_ci_lane.sh <lane>runs every suite in the lane even when one fails, one::group::per suite, an::errorannotation naming each failed suite, and a FAILED list at the end of the lane.run_ci_lane.sh --check, runs even after a red lane) fails if ascripts/tests/test_*file is in no lane, a listed file is gone, or the lane numbers and workflow steps disagree..github/scripts/ci-tests-lanes.pyrebalances the lanes from a run'ssuite-time:log lines.vars.AEON_RUNNER, Blacksmith). Docs: CONTRIBUTING and the aeon skill's ci.md say where new tests go.Measured
test_notify.sh(first suite of lane 7) to exit 3. Lane 7 still ran its other 8 suites, annotatednotify integration tests (exit 3), ended withFAILED suites in lane 7 (1 of 9); the other 9 lanes and the coverage check passed.Risk
A failure shows inside a lane log, not as its own step: the check names the lane, and the annotation plus the FAILED list at the end of the lane name the suite. Suites now run side by side; all use mktemp, and #1237 was green 6/6.
Note: actionlint 1.7.12 does not know
parallel:yet (rhysd/actionlint#693); the workflow lints clean with the group flattened.Need help on this PR? Tag
@codesmith-botwith what you need.