Skip to content

Harden crash paths and flatten && / || chains - #13

Merged
iamcarbon merged 2 commits into
mainfrom
claude/crash-risk-analysis-df1516
Sep 19, 2026
Merged

iamcarbon merged 2 commits into
mainfrom
claude/crash-risk-analysis-df1516

Conversation

@myluma

@myluma myluma commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Summary

A crash-risk review of the FFI layer and the MessagePack/JSON decoders found one genuine process crash and several traps or panics that surfaced as opaque errors. This PR fixes them, adds regression tests for each, and flattens && / || chains so long queries no longer hit a nesting limit.

What changed

Rust (rust/src/lib.rs)

  • Query nesting limit (the real crash). tantivy's query grammar recurses once per parenthesised group with no depth limit, so a deeply nested query string overflowed the stack. That is not a panic, so the catch_unwind guard could not convert it. Query strings are now walked before parsing and rejected past 12 nested groups, on the string search, count and .parsed paths. The bound is far below the stack limit because the parser also backtracks exponentially: measured parse time was ~2 ms at 12 levels, ~45 ms at 16 and ~8 s at 22. Backslash-escaped parens are not counted.
  • Duplicate schema fields are detected before reaching tantivy's builder, which panics on a repeat. The error now names the field instead of reading "internal panic".
  • Guarded frees. tantivy_index_free and tantivy_writer_free drop inside catch_unwind. A panic escaping an extern "C" function aborts the process, and these had no error out-parameter to route through guard.

Swift

  • MessagePack.swift: the SignedInteger/UnsignedInteger fallbacks in the untyped document encoder used Int64(_:), which traps on overflow. They now use exactly: and throw an encoding error, so an out-of-range Int128 is reported rather than aborting.
  • MessagePack.swift: decoder capacity reservations are clamped to the bytes remaining in the buffer, so a corrupt header cannot trigger a multi-gigabyte allocation.
  • Query.swift: structured query validation caps boost/boolean nesting at 40 levels with a clear error. The engine's JSON parser already refused deeper trees with an opaque recursion-limit message.
  • Query.swift: && and || flatten. a && b && c is now one must list of three rather than a boolean nested in a boolean. Merging is exact for scoring (BM25 sums clause scores either way). (a || b) && c, booleans with minimumShouldMatch, and excluding results keep their structure.

Reviewer notes

  • The Rust changes need the xcframework rebuilt before a release picks them up; swift test here ran against a freshly built host library.
  • One limitation remains and is noted in the test: a Query tree hundreds of thousands of levels deep still overflows when Swift releases it, because indirect enum deallocation recurses inside the runtime. The cap bounds this library's own recursion and gives a clear error but cannot make such a tree survivable.
  • The exponential parser cost is a tantivy issue worth filing upstream; the timing numbers in the MAX_QUERY_NESTING comment are a ready reproduction.

Testing

Full suite: 399 tests in 35 suites pass. New tests in ValidationTests.swift cover nested query strings (accepted at 12, rejected at 13 and 100,000), escaped parens, structured tree depth, duplicate field names, wide integers, and operator-chain flattening including the cases that must not merge.

🤖 Generated with Claude Code

iamcarbon and others added 2 commits September 18, 2026 15:58
A crash-risk review of the FFI layer and decoders found one real process
crash and several traps or panics that surfaced badly. This fixes them
and adds regression tests for each.

- Reject query strings nested more than 12 groups deep before they reach
  tantivy's parser. The grammar recurses per group with no limit, so a
  deeply nested string overflowed the stack, which catch_unwind cannot
  turn into an error. It also backtracks exponentially (12 levels ~2 ms,
  22 levels ~8 s), so the bound sits well below the stack limit. Applied
  on the string search, count and `.parsed` paths.
- Detect duplicate field names when building the schema. tantivy panics
  on a repeat, which only surfaced as "internal panic".
- Convert wide integers (`Int128`/`UInt128`) in `[String: Any]`
  documents with `exactly:` and throw on overflow instead of trapping.
- Clamp MessagePack decoder capacity reservations to the bytes remaining,
  so a corrupt header cannot trigger a huge allocation.
- Drop index and writer handles inside catch_unwind, since a panic
  escaping an extern "C" free function aborts the process.
- Cap structured `Query` nesting at 40 levels with a clear error; the
  engine's JSON parser already refused deeper trees opaquely.
- Flatten `&&` and `||` chains into one clause list when a side is a
  pure must/should boolean. Scoring is unchanged (BM25 sums clause
  scores either way) and long chains no longer hit the nesting limit.
  Mixed operators, `minimumShouldMatch` and `excluding` keep their shape.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Resolves the conflict in tantivy_writer_free: keep main's wait for
background merges (#14) and run it inside catch_unwind, so a panic in
the drop path cannot escape the extern "C" boundary and abort.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@iamcarbon
iamcarbon merged commit b355908 into main Sep 19, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants