tests: add volume-delete retry e2e test#1217
Draft
eriknordmark wants to merge 1 commit into
Draft
Conversation
Add an evetestkit suite exercising volume-delete retry behavior on a kubevirt/longhorn EVE-k node: a failed volume destroy must be retained and retried rather than unpublished and leaked, and a permanently-failing delete must be given up on after a bounded number of retries. The suite deploys a tiny VM app, waits for its volume to reach the Created sub-state, then arms a fault on the device (a marker file consumed by EVE's build-tagged volume-delete fault injection) and deletes the app. TestVolumeDeleteRetryRecovers asserts the VolumeStatus stays published in the Deleting sub-state with an error and then disappears once the fault is cleared; TestVolumeDeleteGivesUp asserts it disappears after the retry budget is exhausted while the fault persists. Requires an EVE-k image built with FAULT_INJECTION=y and the volumemgr delete-retry fix (lf-edge/eve#6176). Volume state is read directly off the device (/run/volumemgr/VolumeStatus) because controller-reported state lags. Signed-off-by: eriknordmark <erik@zededa.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An evetestkit suite exercising the volume-delete retry behavior on a kubevirt/longhorn EVE-k node.
TestVolumeDeleteRetryRecovers: deploy a tiny VM app, wait for its volume to reach the Created sub-state, arm a delete fault on the device and remove the app, assert theVolumeStatusis retained in the Deleting sub-state with an error (retried, not leaked), then clear the fault and assert the volume disappears.TestVolumeDeleteGivesUp: same setup but keep the fault armed, and assert the volume disappears after the bounded retry budget is exhausted.Volume state is read directly off the device (
/run/volumemgr/VolumeStatus) because controller-reported state lags.Requirements
FAULT_INJECTION=y— the fault gates from volumemgr: build-tagged fault injection for volume delete eve#6190.Validation
Run on host EVE-k (master base):
TestVolumeDeleteRetryRecoversPASS;TestVolumeDeleteGivesUpPASS.TestVolumeDeleteRetryRecoversFAILS at the "retained in Deleting" assertion — i.e. the test catches the pre-fix leak.Known caveat (draft)
The bounded give-up is
maxVolumeDeleteRetries× the volumemgr gc tick (timer.gc.vdisk/10 s).TestMainlowerstimer.gc.vdiskto its 60 s minimum, but volumemgr only (re)creates its gc ticker at boot, so the fast tick requires an EVE reboot after the setting is applied. Wiring that reboot intoTestMainis a follow-up;TestVolumeDeleteRetryRecoversis event-driven and needs no fast tick.🤖 Generated with Claude Code