Skip to content

Fix three intermittent CI failures, one a sync race at quit - #38

Merged
MHohlios merged 2 commits into
mainfrom
wiffletree/fix-three-intermittent-test-failures-in-ci
Nov 4, 2026
Merged

MHohlios merged 2 commits into
mainfrom
wiffletree/fix-three-intermittent-test-failures-in-ci

Conversation

@MHohlios

@MHohlios MHohlios commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Three tests failed intermittently on GitHub's Ubuntu runner, which would also stop a release. Two were races in the tests. One was a real race in sync: quitting mid-sync could lose the final commit and push. Linear: VYT-263.

Causes and fixes

  • Sync at quit, a product race (sync::tests::a_folder_that_was_the_player_s_own_and_changed_by_sync_waits_again). A quit gives up on the running pass's fetch or push, but its git thread runs on, holding git's locks. The quit's own final pass then fetched beside it and failed with a lock error, shown as "The repo is busy". Repo::off_thread now waits, within its own limit, for a given-up network step's thread to end before starting the next one. Quit is still bounded by quit_wait. The new test git::tests::a_network_step_never_runs_beside_one_given_up_on fails every time without the fix and passes with it.
  • Map restore, a test race (session::tests::a_restore_waits_out_a_merge_landing). The test saved the map version through a second writer before the engine's writer had written room 1, so it could save an empty map. It now calls engine.flush() first. In the app, versions go through the engine's own writer, in order.
  • Credential helper, a test race (git::credentials::tests::a_hanging_helper_is_killed_when_the_limit_is_up). The 200 ms limit could end before the shell ran. The limit is now raised by a watcher once the helper has written its pid. The test still checks that the helper is killed and gone.

Checks

  • cargo fmt --check, clippy (--release --locked -D warnings) and cargo test --workspace --no-fail-fast --locked pass.
  • Under heavy CPU load, the sync test failed 3 times in 1,400 runs before the fix and passed 800 of 800 after it. The restore test failed once in 200 before and passed 200 of 200 after. The helper test didn't reproduce on macOS and passed 200 of 200 after.

Follow-ups

  • A given-up network step now delays the repo's next one until its thread ends, which relies on libgit2's 30 s socket timeout. Confirm the SSH transport honours it (crates/mudling-core/src/git/mod.rs:59).
  • Only fetch and push are serialized; a local commit during an abandoned network thread isn't (crates/mudling-core/src/git/mod.rs:324).

Reviewed with Wiffletree: 1 full review and 1 re-check, 2 rounds (Tests, Claude, Style)

mhohlios added 2 commits November 3, 2026 16:07
A network step given up on at its limit leaves its git thread running,
still holding git's locks. A sync quit during a fetch or push started
the quit's own fetch or push beside it, which failed on the tracking
branch's lock or the remote's ref, so the last sync at quit was lost.
Each network step on a repo now waits for the last one's thread to end,
within its own limit.

Two tests raced themselves:
- The restore test saved a map version through a second writer before
  the engine's writer had put room 1 on the disk, so the version could
  be empty. It now flushes the engine first.
- The hanging helper test started its 200 ms limit before the shell
  helper ran, so a slow start was killed before it wrote its pid. The
  limit now ends once the helper is running.
@MHohlios
MHohlios merged commit babb881 into main Nov 4, 2026
5 checks passed
@MHohlios
MHohlios deleted the wiffletree/fix-three-intermittent-test-failures-in-ci branch November 4, 2026 00:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant