Repository navigation
fix: make baseline E2E hangs fail safely - #608
AlexanderZ-Band wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
There are reliability risks (sub-100ms watchdog deadlines in unit tests and unpinned python interpreter usage in the runner script) that can cause flaky CI or unexpected runtime failures.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR hardens the baseline E2E CI harness against “hung with no progress” failure modes by adding a cross-platform watchdog that terminates a stuck pytest process tree, emits a safe diagnostic (current pytest nodeid only), and keeps the existing fail-closed scorecard/matrix semantics. It also tightens the partial rehydration smoke to require an offline peer-authored, explicitly addressed marker so the test unambiguously exercises agent-scoped /context rehydration.
Changes:
- Add
.github/scripts/watch-progress.pyand wrap baseline E2E pytest runs with a no-output watchdog that writes a safe diagnostic and exits with a distinct code on timeout. - Emit
E2E_PROGRESS nodeid=...from the baseline pytest hooks so the watchdog can identify the currently running test safely. - Update CI wiring to pass watchdog settings/paths and upload watchdog diagnostics as artifacts; refine the partial rehydration smoke’s setup/wording to assert agent-scoped history correctly.
File summaries
| File | Description |
|---|---|
tests/framework_conformance/test_e2e_ci_scripts.py |
Adds behavioral tests for the watchdog utility and ensures the baseline runner fails closed on watchdog timeouts. |
tests/e2e/baseline/smoke/matrix/test_rehydration_partial.py |
Strengthens the scenario to require an offline peer-authored, explicitly mentioned marker to validate /context rehydration unambiguously. |
tests/e2e/baseline/conftest.py |
Emits a safe per-test progress marker (E2E_PROGRESS nodeid=...) for the watchdog to track. |
.github/workflows/e2e.yml |
Wires watchdog env/diagnostic paths and uploads watchdog diagnostics as CI artifacts. |
.github/scripts/watch-progress.py |
New cross-platform watchdog that monitors output, terminates stuck process trees, and writes safe diagnostics. |
.github/scripts/run-baseline-e2e.sh |
Wraps baseline pytest execution in the watchdog and plumbs per-attempt diagnostic output. |
Review details
- Files reviewed: 6/6 changed files
- Comments generated: 6
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| BAND_E2E_SCORECARD_JSON="$scorecard" \ | ||
| python .github/scripts/watch-progress.py \ | ||
| --idle-seconds "$E2E_PROGRESS_DEADLINE_SECONDS" \ | ||
| --diagnostic "$diagnostic" \ | ||
| -- uv run pytest tests/e2e/baseline/ -v -s --no-cov "$@" |
| assert process.stdout is not None | ||
| lines: Queue[str | None] = Queue() | ||
| Thread(target=_read_lines, args=(process.stdout, lines), daemon=True).start() | ||
|
|
| result = _watch_progress( | ||
| tmp_path, | ||
| "import time; print('E2E_PROGRESS nodeid=tests/e2e/test_hang.py::test_hang', flush=True); time.sleep(2)", | ||
| idle_seconds=0.1, | ||
| ) |
| "ATTEMPT2": str(attempts / "two.json"), | ||
| "FINAL": str(tmp_path / "artifacts" / "scorecard-core-ubuntu.json"), | ||
| "PROGRESS_DIAGNOSTIC1": str(diagnostics / "one.json"), | ||
| "PROGRESS_DIAGNOSTIC2": str(diagnostics / "two.json"), | ||
| "E2E_PROGRESS_DEADLINE_SECONDS": "0.1", | ||
| }, |
| #!/usr/bin/env python3 | ||
| """Run a command and fail it if its safe progress signal stops advancing.""" |
| require_dep(dep, settings) | ||
|
|
||
|
|
||
| def pytest_runtest_logstart(nodeid: str, location: tuple[str, int | None, str]) -> None: |
|
Closing without merging. On review, this treats the symptom (an E2E leg silently hard-hanging) rather than the actual defect: the Claude Agent SDK's subprocess transport not reliably dying under raw asyncio cancellation (the ticket's own item 4, adapter-level turn containment, was deferred pending a cleanup contract for exactly that reason). Checked GitHub's run history: the underlying hang has recurred on |
Summary
Verification
uv run pytest tests/framework_conformance/test_e2e_lane_drift.py tests/framework_conformance/test_e2e_ci_scripts.py -q --no-covuv run ruff check ...uv run pyrefly check .github/scripts/watch-progress.pybash -n .github/scripts/run-baseline-e2e.shNo live E2E was run.
Closes INT-1378