Skip to content

Reap leftover EVE VM before running eden tests#1216

Merged
rene merged 1 commit into
lf-edge:masterfrom
europaul:fix/reap-orphaned-eve-vm-before-test
Jul 17, 2026
Merged

Reap leftover EVE VM before running eden tests#1216
rene merged 1 commit into
lf-edge:masterfrom
europaul:fix/reap-orphaned-eve-vm-before-test

Conversation

@europaul

Copy link
Copy Markdown
Contributor

Problem

Self-hosted runners were getting wedged: several kratos-runner-eden-* runners had a leftover qemu-system-x86_64 (an EVE VM) holding the console port 17777, so every subsequent eden test job on that runner failed at eden start with the port already in use.

Tracing it back, the orphaned VMs all came from the Smoke suite's eve_restart test in jobs that were cancelled mid-test (e.g. a newer PR-Gate run superseding an in-flight one).

Why the existing cleanup doesn't handle it

The Clean up after test step in this composite action (./eden stop etc.) is guarded by if: always():

  • always() runs on success and failure, but a composite action's remaining steps are not executed when the job is cancelled — the whole action is torn down mid-run. So on cancellation the VM is never stopped.
  • It also can't be reaped after the fact: eden tracks the VM through a context stored under the ephemeral HOME of the cancelled test run, which is gone. A later job's eden stop / make clean finds no context and skips stopping the process. Verified on the stuck runners — running both eden stop and make clean were no-ops and the qemu kept running; only freeing the port directly reaped it.

Fix

Add a pre-test step that frees port 17777 and cleans the workspace before Setup Environment, so a leftover from a previous job can't block the next one. Killing by the well-known EVE console port is surgical — it only targets an orphaned EVE VM, not any unrelated qemu.

- name: Clean up before test
  run: |
    fuser -k 17777/tcp || true
    make clean || true

Cancelling a job mid-test leaves its EVE VM running, holding the console
port (17777). The next job scheduled on the same self-hosted runner then
fails to start EVE because the port is already taken.

Why the existing "Clean up after test" step does not handle this:

- It lives inside this composite action and is guarded by `if: always()`.
  always() runs on success and failure, but a composite action's remaining
  steps are not executed when the job is cancelled - the whole action is
  torn down mid-run. So on cancellation the VM is never stopped.

- It cannot be reaped after the fact either: eden tracks the VM through a
  context stored under the ephemeral HOME of the cancelled test run, which
  is gone. A later job's `eden stop` / `make clean` finds no context and
  skips stopping the process (verified on the stuck runners - both were
  no-ops and the qemu kept running).

Add a pre-test step that frees port 17777 directly and cleans the
workspace, so a leftover from a previous job cannot block the next one.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Paul Gaiduk <paulg@zededa.com>
@europaul
europaul force-pushed the fix/reap-orphaned-eve-vm-before-test branch from 7e404c0 to 0eee61c Compare July 17, 2026 09:51
Comment thread .github/actions/run-eden-test/action.yml
@europaul

Copy link
Copy Markdown
Contributor Author

@rene we should add "stable" label to Eden github repo. for example for PRs like this one

@rene rene added the stable Should be backported to stable release(s) label Jul 17, 2026
@rene

rene commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

@rene we should add "stable" label to Eden github repo. for example for PRs like this one

yes, I've created the label and added to this PR as well.

@rene
rene merged commit cb866fc into lf-edge:master Jul 17, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stable Should be backported to stable release(s)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants