Repository navigation
fix(lancedb): treat a lance spill failure as retryable - #463
Merged
Merged
Conversation
The cascade worker retries ExternalServiceError with backoff and files any other exception as unrecoverable. lancedb raises 'LanceError(IO): Execution error: Spill has sent an error' -- DataFusion's sort/merge spill to the OS temp dir failing mid-query -- as a bare RuntimeError, so every occurrence marked an md file permanently failed. On the Windows soak box it fired 187 / 180 / 34 times across three runs under nine concurrent clients, never on an idle box, and the same rows project fine on retry; about 200 files per run were left for a manual 'cascade fix'. Translate that one phrase into VectorStoreBusyError inside the read deadline wrapper (the observed path: worker -> find/delete by md_path -> repository -> lancedb query). Other IO errors keep propagating unchanged. Tests: the phrase is translated, an unrelated IO error is not; removing the marker fails the translation test. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
gloryfromca
enabled auto-merge (squash)
September 24, 2026 08:54
arelchan
self-requested a review
September 24, 2026 08:59
arelchan
approved these changes
Sep 24, 2026
This was referenced Sep 24, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The cascade worker retries
ExternalServiceErrorwith backoff and files any other exception as unrecoverable (mark_failed(retryable=False), surfaces incascade fix). lancedb raisesLanceError(IO): Execution error: Spill has sent an error— DataFusion's sort/merge spill to the OS temp dir failing mid-query — as a bareRuntimeError, so every occurrence marked an md file permanently failed.On the Windows soak box (ThinkPad, nine concurrent clients) it fired 187 / 180 / 34 times across the three soak runs, never on an idle box, and the same rows project fine on the next attempt; roughly 200 files per run were left needing a manual
cascade fix. The run-1 report attributed the unrecoverable count to fuzzed malformed markdown only; this class was hiding under it.Fix: translate that one phrase into
VectorStoreBusyErrorinside the read deadline wrapper (LanceRepoBase._deadline) — the observed path is worker →find_by_md_path/delete_by_md_path→ repository → lancedb query. Other IO errors (No space left, missing file) propagate unchanged and stay permanent.Area
Verification
tests/unit/test_core/test_persistence/test_lancedb/test_transient_execution_errors.py: the phrase becomesVectorStoreBusyError; an unrelated IO error does not; the marker predicate. Emptying the marker tuple fails the translation test.test_core/test_persistence/test_lancedb+test_cascade: green.ruffgreen.unrecoverable_totalshould drop to the fuzz-only baseline).Checklist
Notes for Reviewers
Why not a broader
LanceError(IO)match: disk-full and vanished-file errors carry the same prefix and must stay permanent. Why not in_lockedtoo: the write path was not where the error was observed; widening is a one-line follow-up if a soak shows it there.🤖 Generated with Claude Code