Skip to content

fix(tests): await parallel cache consumers before assertions - #3352

Merged
Daniel Meppiel (danielmeppiel) merged 1 commit into
mainfrom
danielmeppiel-ci-cache-worker-completion
Oct 22, 2026
Merged

Daniel Meppiel (danielmeppiel) merged 1 commit into
mainfrom
danielmeppiel-ci-cache-worker-completion

Conversation

@danielmeppiel

@danielmeppiel Daniel Meppiel (danielmeppiel) commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

fix(tests): await parallel cache consumers before assertions

Description

TL;DR

Wait for both real Git consumers to finish before the shared-cache regression test inspects their results. The previous timed joins could return with a consumer still running and report a correctly materialized package as missing. Keep both consumers, their barrier, real Git operations and all content/error assertions; also select this case in the existing Windows compatibility gate.

Important

This is a test-lifetime correction, not a production-cache concurrency fix or a timeout increase. The failed main-push run stays failed; a separate successful execution does not erase it.

Problem (WHY)

  • Actual public-main Windows job 119842360862 failed this one case with AssertionError: assert 'X' in ['Y']. The unit step finished with 1 failed, 24835 passed, 624 skipped, 21 xfailed.
  • A controlled execution of the unchanged case reproduced that exact assertion after real checkout succeeded and the X file existed: join(timeout=10) returned while thread_a was alive. The worker subsequently completed; no production-cache failure was needed to reproduce the false result.
  • [!] The hosted traceback did not capture thread liveness. The controlled trace establishes the fixture defect, not an invented observation about that hosted thread.

The investigation follows Agent Skills' "Read agent execution traces, not just final outputs.": preserve the failed run, inspect the real boundary, and distinguish an incomplete observation from missing content.

Approach (WHAT)

  • Submit both existing consumers to a two-worker executor before collecting either future.
  • Wait for both results before the unchanged error/content assertions and cache cleanup.
  • Preserve the five-second barrier and actual local/shared Git clone and checkout commands.
  • Add windows_compat to this case only; do not broaden the entire module or edit CI selection.

Implementation (HOW)

File Bounded change
tests/unit/deps/test_shared_clone_cache.py Replace unchecked timed joins with executor-owned completion. Keep both worker bodies and all three final assertions; add the existing scheduling marker.
docs/src/content/docs/contributing/integration-testing.md Explain why concurrent fixtures must await completion before observing results or cleaning resources.

No production code, credentials, cache keys, package references, workflow caps, marker expressions, security controls, dependencies or release behavior change. No CHANGELOG entry is added for this internal test correction.

Diagrams

The highlighted region owns both worker lifetimes; the real parallel materialization and final assertions remain unchanged.

sequenceDiagram
    participant T as Cache regression test
    participant E as ThreadPoolExecutor
    participant A as thread_a
    participant B as thread_b
    rect rgb(255, 247, 200)
        Note over T,E: NEW: own both worker lifetimes
        T->>E: Submit both consumers
        E-->>T: Two futures
        T->>E: Collect both future results
        par Existing consumer A
            E->>A: Run thread_a
            A->>A: Barrier then real Git clone and checkout
            A-->>E: Return
        and Existing consumer B
            E->>B: Run thread_b
            B->>B: Barrier then real Git clone and checkout
            B-->>E: Return
        end
        E-->>T: Both consumers completed
    end
    T->>T: Assert no errors and X plus Y present
    T->>T: Delete temporary bare cache
Loading

Trade-offs

  • Await actual completion rather than guessing a larger timed-join budget. This test checks content/concurrency, not a latency threshold; the existing barrier and workflow limits remain unchanged.
  • The slow-worker and missing-content controls are retained diagnostic artifacts, not added sleeps or mutation switches in the normal suite. They exercise real Git and do not replace the cache with a mock.
  • Keep production cache logic untouched: the demonstrated defect is in the fixture's observation lifetime. No separate retry is offered as proof of repair.

Benefits

  1. Both consumer futures complete before any final result assertion.
  2. A real missing-X result still fails the original content assertion.
  3. The existing PR-time Windows gate now selects this exact cross-platform regression case.

Issue and approved scope

Issue: None. This is a separate bounded CI-recovery carrier under the maintainer's direct task-specific exception, not a claim of accepted issue status.

Human scope-approval comment: No GitHub approval comment is claimed. The standing direct instruction is: "you do it, and do not mandate for these fixes to require an issue and approval, it's not needed. get CI working ASAP". The coordinator explicitly returned exclusive ownership of this newly observed failure.

Does this PR complete the issue, or what remains? No issue is closed. Scope is exactly the two files above. The wider main-CI/nonpublishing-validation mission remains separate; delivered PRs and their historical evidence are not reopened or relabelled.

Type of change

  • Bug fix
  • New feature
  • Documentation
  • Maintenance / refactor

Testing

  • Tested locally
  • All existing tests pass
  • Added tests for new functionality (if applicable)

No new product functionality is introduced. The existing regression is repaired and gains Windows-gate coverage. Local claims remain limited to the focused suite and explicit controls below; this carrier's exact-head hosted checks are now terminal.

Validation

Exact candidate: 9f7c0108bd8012662dd087d711558a6291544e10, sole parent actual public main abd363ff7bfabbc7984e1bad8cdff373f44219fa. Both paths were frozen before qualification; clean-source guards passed before and after every execution.

Execution Result
Unchanged abd case, delay consumer A for 12 seconds after successful real checkout 1 intended failure, same missing-X assertion; timed join returned with A alive, X already existed, all workers subsequently drained
Exact 9f7c010 case, identical delayed-checkout control 1 passed in 13.15s; workers drained before the result
Exact 9f7c010 case, remove X after successful real checkout 1 intended failure in 1.02s, original missing-X assertion
Restored focused cache module 45 passed in 4.41s, zero failures/errors/skips
Canonical lint/static contract on exact 9f7c010 Passed, every command exit 0; YAML-I/O, portable-path and 2100-line guards passed
Fresh canonical-owner detector, abd..9f7c010 0 touched owners; no application-owner witness is fabricated
CI 39790701019, exact 9f7c010 16 workflow jobs succeeded; the separately attached eligibility advisory is neutral
Windows compatibility job 119865653070 Repaired case explicitly PASSED; 863 passed, 2 skipped, 36088 deselected in 519.19s
Merge gate 39790701199 Succeeded; all 26 rollup entries terminal: 23 success, 2 neutral, 1 skipped

Both actual CodeQL analyses succeeded. The imposed-native CodeQL check remains neutral/missing under the existing explicit maintainer exception, not passed; docs deployment was skipped. Independent bounded source review found no in-scope blocker; Copilot's review had no inline findings. Docs-sync returned deterministic L0 no_change, with no panel spawned.

Warning

This carrier does not clear the wider mission. The separate actual-abd full validation 39784442659 has 10 Linux ARM integration failures and 6 Intel macOS integration failures; Linux x86 integration was cancelled and Release Validation skipped because of failed dependencies. Those are genuine recovery blockers, not allowed publication skips or results for this head.

Exact commands and retained output

The session-only diagnostic harness and its source-bound traces are retained with the maintainer's evidence packet; they are not runtime dependencies or files added to this PR.

FILES=/Users/danielmeppiel/.copilot/session-state/067b35fc-52fb-4db4-b316-c606d23b4b34/files
GATE=/Users/danielmeppiel/.copilot/session-state/f79e464c-180e-4be3-9056-0773aa679a1a/files/campaign-em/run-heavy.py

Delayed control, exit 0 on the candidate (the same control exited 1 on unchanged abd):

python3 "$GATE" --lane main-ci-recovery --timeout 1800 -- uv run --frozen --extra dev python -c "import runpy; runpy.run_path('$FILES/cache_worker_completion_probe.py', run_name='__main__')" --delay 12 --report "$FILES/cache-worker-fixed-delay.json" --junitxml "$FILES/cache-worker-fixed-delay.xml" --basetemp "$FILES/cache-worker-fixed-delay-tmp"
1 passed in 13.15s

Missing-content control, expected exit 1:

python3 "$GATE" --lane main-ci-recovery --timeout 1800 -- uv run --frozen --extra dev python -c "import runpy; runpy.run_path('$FILES/cache_worker_completion_probe.py', run_name='__main__')" --delay 0 --remove-x --report "$FILES/cache-worker-missing-content.json" --junitxml "$FILES/cache-worker-missing-content.xml" --basetemp "$FILES/cache-worker-missing-content-tmp"
E   AssertionError: assert 'X' in ['Y']
1 failed in 1.02s

Restored module, exit 0:

python3 "$GATE" --lane main-ci-recovery --timeout 1800 -- uv run --frozen --extra dev python -m pytest tests/unit/deps/test_shared_clone_cache.py -p no:randomly -p no:cacheprovider -q --junitxml="$FILES/cache-worker-focused.xml" --basetemp="$FILES/cache-worker-focused-tmp"
45 passed in 4.41s

Each following canonical command passed with exit 0:

uv run --frozen --extra dev ruff check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
uv run --frozen --extra dev ruff format --check src/ tests/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
uv run --frozen --extra dev python -m pylint --disable=all --enable=R0801 --min-similarity-lines=10 --fail-on=R0801 src/apm_cli/ scripts/lint_architecture_boundaries.py scripts/architecture_linter/ scripts/windows_native_symlink_probe_entry.py
bash scripts/lint-auth-signals.sh
bash scripts/lint-architecture-boundaries.sh
uv run --frozen python scripts/check_test_assertions.py
uv run --frozen python scripts/check_exact_test_duplicates.py

The platform-neutral equivalents of CI's three grep/awk guards passed on their exact covered paths. The diagram also passed the installed Mermaid CLI 12.0.0.

Scenario Evidence

# Scenario (user promise) Principle(s) Test(s) proving it Type
1 Contributors can validate two real parallel package consumers without mistaking a still-running consumer for lost content. OSS / community-driven tests/unit/deps/test_shared_clone_cache.py::TestBareCacheRaceCondition::test_parallel_different_subdirs_both_succeed (regression trap for the linked failed main job; delayed-completion and missing-content controls above) integration

This is a real-filesystem/local-Git test located under tests/unit; the table does not claim a user CLI flow or a mocked production-cache proof.

How to test

  • Run uv run --frozen --extra dev python -m pytest tests/unit/deps/test_shared_clone_cache.py -q; expect all 45 cases to pass.
  • Run uv run --frozen --extra dev python -m pytest tests/unit/deps/test_shared_clone_cache.py -m windows_compat -q; expect the repaired case to be selected and pass.
  • Inspect the case: both submissions precede result collection, the barrier remains, and the original error/X/Y assertions are unchanged.
  • Check this PR's Windows compatibility and ordinary CI results on the exact published head; an older main run or another head is not clearance.

Spec conformance (OpenAPM v0.1)

If this PR changes behaviour that an OpenAPM v0.1 req-XXX covers,
confirm the three-step ritual in the
development guide:

  • Spec edit: docs/src/content/docs/specs/openapm-v0.1.md updated
    (new/changed <a id="req-XXX"></a> anchor + prose + Appendix C
    row).
  • Manifest edit: docs/src/content/docs/specs/manifests/openapm-v0.1.requirements.yml
    updated.
  • Test edit: a @pytest.mark.req("req-XXX") test under
    tests/spec_conformance/ added or extended.
  • CONFORMANCE.{md,json} regenerated via
    uv run --extra dev python -m tests.spec_conformance.gen_statement
    and committed.
  • N/A -- this PR does not change OpenAPM-observable behaviour.

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 067b35fc-52fb-4db4-b316-c606d23b4b34
Copilot AI balanced review requested due to automatic review settings October 22, 2026 13:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The bounded test correction waits for both workers without weakening concurrency or content checks, and the documentation matches the change.

No open issues 🎉

What changed in this PR

This PR fixes a shared-cache regression test that could inspect results before both Git consumers had finished, without changing production cache behavior.

Changes:

  • Replace timed thread joins with executor-managed completion while preserving parallel execution and existing assertions.
  • Select the test in the existing Windows compatibility gate.
  • Document worker-completion requirements for concurrent fixtures.
File Description
tests/​unit/​deps/​test_shared_clone_cache.py Await both consumers and add the windows_compat marker.
docs/​src/​content/​docs/​contributing/​integration-testing.md Explain when concurrent fixtures must await workers.

Review effort: Balanced

@danielmeppiel

Daniel Meppiel (danielmeppiel) commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator Author

Docs sync advisory

Verdict: no_change * Pages affected: 0 * LLM calls: 0/15 * Took: not timed (deterministic path gate)

Both changed paths match the canonical test/docs no-impact gate; no user-facing CLI, policy, dependency format, primitive or TOC surface changed. The contributor guide already explains worker completion before result inspection and cleanup.

This is a direct L0 path classification, not a panel review or approval.


Generated by docs-sync. This comment is AI-generated and may contain errors.

Merge Master assessment: ship_now

Qualified exact head 9f7c0108bd8012662dd087d711558a6291544e10 against actual public main abd363ff7bfabbc7984e1bad8cdff373f44219fa. This is a separate technical landing assessment, not a new panel or formal approval. The genuine independent bounded source review and current-head Copilot review have no outstanding in-scope findings.

All 50 artifact seals, the full completion schema and strict terminal owner-v2 verification passed; zero application owners are touched. Actual source-session requests and terminal receipts match the retained evidence. The unchanged delayed-worker control fails, the identical control on this head passes, missing X still fails the original content assertion, and the restored focused module passes all 45 cases. Canonical lint/static evidence is exact-head and terminal. Both real Git consumers, their barrier and all substantive assertions remain intact.

Own-head CI and merge gate are successful, including explicit Windows execution of the repaired node. All 28 currently observed checks are terminal: 25 success, two neutral and one intended deployment skip. The added entries since the source's 26-check snapshot are metadata-only eligibility publications. Actual CodeQL analyses succeeded; the imposed neutral/missing native CodeQL comparison remains an explicit exception, not passed analysis. The original failing run and both failed evidence guards remain preserved.

The author released all #3352 write/process surfaces. Fresh identity, unique carrier, empty closing references, body, live reviews, current-main compatibility and admin bypass capability were verified. This bounded issue-less CI repair closes no issue.

The wider CI mission remains open. Full-run 39784442659 has unresolved required integration failures, a cancelled Linux leg and an unwanted Release Validation skip. They are separate from this repair, are not waived, and are not evidence for this head. No release, tag, deployment or additional dispatch is authorized by this assessment. Final live guards still precede the serialized landing.


Generated by autopilot-pr-merge-worker. This comment is AI-generated and may contain errors.

@danielmeppiel
Daniel Meppiel (danielmeppiel) merged commit 33f738e into main Oct 22, 2026
28 checks passed
@danielmeppiel
Daniel Meppiel (danielmeppiel) deleted the danielmeppiel-ci-cache-worker-completion branch October 22, 2026 14:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants