Reap leftover EVE VM before running eden tests#1216
Merged
rene merged 1 commit intoJul 17, 2026
Conversation
Cancelling a job mid-test leaves its EVE VM running, holding the console port (17777). The next job scheduled on the same self-hosted runner then fails to start EVE because the port is already taken. Why the existing "Clean up after test" step does not handle this: - It lives inside this composite action and is guarded by `if: always()`. always() runs on success and failure, but a composite action's remaining steps are not executed when the job is cancelled - the whole action is torn down mid-run. So on cancellation the VM is never stopped. - It cannot be reaped after the fact either: eden tracks the VM through a context stored under the ephemeral HOME of the cancelled test run, which is gone. A later job's `eden stop` / `make clean` finds no context and skips stopping the process (verified on the stuck runners - both were no-ops and the qemu kept running). Add a pre-test step that frees port 17777 directly and cleans the workspace, so a leftover from a previous job cannot block the next one. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Paul Gaiduk <paulg@zededa.com>
europaul
force-pushed
the
fix/reap-orphaned-eve-vm-before-test
branch
from
July 17, 2026 09:51
7e404c0 to
0eee61c
Compare
rene
reviewed
Jul 17, 2026
rene
approved these changes
Jul 17, 2026
Contributor
Author
|
@rene we should add "stable" label to Eden github repo. for example for PRs like this one |
Contributor
yes, I've created the label and added to this PR as well. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Self-hosted runners were getting wedged: several
kratos-runner-eden-*runners had a leftoverqemu-system-x86_64(an EVE VM) holding the console port 17777, so every subsequent eden test job on that runner failed ateden startwith the port already in use.Tracing it back, the orphaned VMs all came from the Smoke suite's
eve_restarttest in jobs that were cancelled mid-test (e.g. a newer PR-Gate run superseding an in-flight one).Why the existing cleanup doesn't handle it
The
Clean up after teststep in this composite action (./eden stopetc.) is guarded byif: always():always()runs on success and failure, but a composite action's remaining steps are not executed when the job is cancelled — the whole action is torn down mid-run. So on cancellation the VM is never stopped.eden stop/make cleanfinds no context and skips stopping the process. Verified on the stuck runners — running botheden stopandmake cleanwere no-ops and the qemu kept running; only freeing the port directly reaped it.Fix
Add a pre-test step that frees port 17777 and cleans the workspace before
Setup Environment, so a leftover from a previous job can't block the next one. Killing by the well-known EVE console port is surgical — it only targets an orphaned EVE VM, not any unrelated qemu.