Skip to content

Guard every string crossing the FFI, not just the query - #8

Merged
iamcarbon merged 1 commit into
mainfrom
guard-ffi-string-boundary
Jul 31, 2026
Merged

iamcarbon merged 1 commit into
mainfrom
guard-ffi-string-boundary

Conversation

@myluma

@myluma myluma commented Jul 31, 2026

Copy link
Copy Markdown
Collaborator

An interior NUL truncates a Swift string at the C boundary. validateQueryString already guarded the query string against exactly this; every other string reaching the engine was unguarded. Field names did truncate, and when the truncated prefix is itself a valid field name the call silently operates on that field instead of erroring.

The one with teeth

Demonstrated before fixing, on an index with title and titleSlug:

deleteDocuments(field: "title\0Slug", equals: "red")  ->  2 docs -> 1 doc
deleteDocuments(field: "titleSlug",   equals: "red")  ->  2 docs -> 2 docs   (correct: no match)

titleSlug values are red-slug/blue-slug, so the requested field matches nothing — yet a document is deleted, because the name truncated to title. The same silent substitution applied to search(fields:) (returned a hit from title), highlight: (highlighted title), orderBy:, and aggregate(), where json + "\0garbage" ran the parseable prefix and dropped the rest without complaint.

Fixed by generalizing the existing guard into Index.validateNoInteriorNUL / validateFieldName and applying it at every boundary: fields, highlight, orderBy.field, boosts keys, deleteDocuments(field:), aggregate, termCounts. The comma check comes along for free — field names join into a CSV, so an embedded comma widens the field set the same way a NUL narrows it.

The structured Query path was already safe: JSON-escaping preserves the NUL, so the engine rejects the field outright rather than substituting one.

Two smaller ones found alongside

Float decode overflowed to infinity. Float(try Convert.double(...)) at three call sites, so a stored 1e300 decoded as inf in the keyed, unkeyed, and single-value containers — while Double kept the value and every integer conversion range-checks via exactly: and throws. Silently substituting infinity for a real measurement is the worst of the three options. Added Convert.float with a range check.

Missing argument validation. writer(heapSize: -1) wrapped to usize::MAX and surfaced as the engine's per-thread arena-cap error, which never mentions the value passed; termCounts(limit: -1) surfaced as an opaque serde expected u32. Both now throw .encoding, as search does for limit.

Tests

234 -> 256. Both regression tests were verified to fail without their fix — the delete test reports documentCount → 1, the float test Scalar(big: inf) — so they aren't vacuous.

Coverage also added for behaviour that was already correct but untested:

  • write's documented all-or-nothing guarantee, and that the discarded writer releases the directory lock (nothing covered this; a leaked lock would break every subsequent write after one failed block)
  • UInt64.max round-tripping through store and term query, the exactness FieldValue documents
  • sub-second dates truncating rather than rounding, and date terms still matching across that granularity
  • the silent-empty traps: allOf([]) matches nothing, minimumShouldMatch above the clause count matches nothing
  • minWordLength / maxWordLength / maxDocFrequency — the only public symbols with no test reference at all

Left unfixed

Engine error messages containing a NUL are lost, degrading to a bare tantivy: error (CString::new fails on the message in rust/src/lib.rs). That needs a Rust rebuild and release, and the guards here make it much harder to reach, so it's better folded into the next release.

🤖 Generated with Claude Code

An interior NUL truncates a Swift string at the C boundary. validateQueryString
covered the query string; every other string reaching the engine was unguarded,
and truncating to a prefix that is itself a valid field name silently operates
on that field instead:

  deleteDocuments(field: "title\0Slug", equals: "red")  ->  2 docs -> 1 doc
  deleteDocuments(field: "titleSlug",   equals: "red")  ->  2 docs -> 2 docs

The requested field matches nothing, yet a document is deleted. Same
substitution on search(fields:), highlight:, orderBy:, and aggregate(), where
trailing bytes after a NUL are silently dropped.

Also fixes two smaller ones found alongside it:

  - Float decode overflowed to infinity. Float(1e300) is +inf, so an
    out-of-range f64 decoded as an infinity while Double kept the value and
    every integer conversion range-checks and throws.
  - writer(heapSize:) and termCounts(limit:) accepted negatives, surfacing as
    the engine's arena-cap error or an opaque serde message rather than saying
    what was passed.

Adds coverage for behaviour that was already correct but untested: write's
all-or-nothing guarantee and its lock release, UInt64.max round-tripping,
sub-second dates truncating rather than rounding, the empty-boolean and
minimumShouldMatch match-nothing cases, and the three MoreLikeThisOptions
knobs no test referenced.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@iamcarbon
iamcarbon merged commit fd2cfb9 into main Jul 31, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants