Repository navigation
CI: correctness on PRs, performance daily - #33
Merged
MHohlios merged 5 commits intoNov 3, 2026
Merged
Conversation
added 5 commits
November 3, 2026 09:22
…viour out Each carries #[ignore = "timing: runs daily"] and a name starting timing_. Where a test also checked behaviour, the behaviour stays its own test. Plugin checks gain own_tests_except_cost, so a plugin's transcript and hooks are tested without judging its cost a line.
ci.yml drops the replay benchmark and the comparison with the base, and runs cargo test with --no-fail-fast. daily.yml runs the benchmark, the comparison with the last measured commit, and the timing tests, on main. docs/ci.md says which is which.
The macOS runners' bash is 3.2, which has no mapfile, so a failed first attempt ended the daily timing job instead of retrying. The build job's check name goes back to what a ruleset may require.
mudling plugin test skips the cost when MUDLING_SKIP_COST_CHECK is set, and a link fetch has a test-only way to skip it, so pull request tests no longer fail on a slow runner. Timing tests keep judging cost. Avatar's transcript test now lives in the CLI tests, so the avatar crate's copy goes.
…--ignored GitHub notifies only the user who created the workflow or last changed its cron line; everyone else sees a failure on the Actions tab. An unfiltered cargo test -- --ignored rewrites a committed fixture and reaches the GPU and the live registry. The Aardwolf test's comment now says it skips the cost.
MHohlios
force-pushed
the
wiffletree/ci-correctness-on-prs-performance-daily
branch
from
November 3, 2026 14:24
f36e92a to
5792fa1
Compare
MHohlios
deleted the
wiffletree/ci-correctness-on-prs-performance-daily
branch
November 3, 2026 15:19
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PRs now run correctness checks only. The replay benchmark, its comparison and every test whose result depends on machine speed move to a daily run on main. Linear: VYT-259.
What runs where
ci.yml): fmt and clippy, the build,cargo test --workspace --no-fail-fast, the macOS--synthetic --ax-check, and the iOS core build. The job keeps its name "Build, test and bench ()" so a future required check still matches. "Benchmark against the base branch" is gone.daily.yml): the replay benchmark on macOS, Windows and Ubuntu;mudling-bench compareagainst the commit the previous daily run measured, or the latestv*tag; and the timing tests, best of three, retrying only the tests that failed.release.yml): unchanged; it still runsci.ymlfirst.Timing tests
Ten tests are marked
#[ignore = "timing: runs daily"]and namedtiming_*. A test that also checked behaviour was split, so the behaviour half still runs on every PR. The plugin cost check is skipped in behaviour tests throughown_tests_except_cost,fetch_link_except_costand, for the CLI tests,MUDLING_SKIP_COST_CHECK. The app's own link install still judges cost.Run them locally with
cargo test --workspace -- --ignored timing_on a quiet machine.docs/ci.mdexplains both runs.GitHub emails a scheduled run's failure only to whoever last changed its
cronline; others see it on the Actions tab.Follow-ups
timing_if one flakes (crates/mudling-core/src/plugin/matcher.rs:316).crates/mudling-core/tests/luau_hooks.rs:268)..github/workflows/daily.yml:5).MUDLING_SKIP_COST_CHECKships in the released CLI; move it to a test-only path (crates/mudling-cli/src/check.rs:24).Reviewed with Wiffletree: 1 full review and 1 re-check, 3 rounds (Tests, Claude, Style)