[myluma]

What changed

Three additions so a host can query stemmed fields with typed terms instead of the query parser, and collapse its _raw mirror fields into one field that answers exact, stemmed, prefix and phrase queries.

Index.analyze(_:with:) — Release A

Runs a field's analyzer on a string and returns its Tokens (text, position, UTF-8 byte offsets). New tantivy_index_analyze export; reads the index's own tokenizer manager, so registered analyzers (lowercase, en_stem_keep) resolve as well as the built-ins. Token has a public init so callers can add synonyms to an analyzed stream.

Analyzer.englishKeepingSurface (en_stem_keep) — Release B

Tantivy's en_stem chain (SimpleTokenizer → RemoveLong(40) → LowerCaser) with the stemmer swapped for a new KeepStemFilter (rust/src/keep_stem.rs): each word is emitted as itself, then as its stem at the same position when the stem differs (designers → designers + design; poster once). Later positions are not shifted, so phrases work across the doubled position.

Query.multiPhrase — synonym-aware phrases

A phrase where each position accepts any one of several terms (Lucene's MultiPhraseQuery): .multiPhrase("f", [["graphic"], ["designers", "design"]]). .multiPhrase("f", tokens:) takes analyze output directly, grouping same-position tokens into alternatives; PhrasePosition(offset:terms:) for explicit gaps.

Things a reviewer should know

• .parsed on an en_stem_keep field is an exact-surface match. Tantivy's parser puts same-position tokens into one PhraseQuery requiring all of them, so designers only matches the literal word — the stemming is silently lost. The enum doc comment says to use typed queries from analyze; parsedQueriesLoseStemming pins the behaviour.
• multiPhrase is built on RegexPhraseQuery, with each position an escaped literal alternation (designers|design). Tantivy keeps the true multi-phrase machinery crate-private. Alternatives are anchored and literal (c++, well-known, c|cxx tested).
• Scoring rescale. RegexPhraseQuery takes each position's idf from the pattern text, which is never an indexed term, so every position looked maximally rare. The weight is rescaled so each position counts as its most frequent alternative (Lucene SynonymQuery semantics): adding a synonym never raises a score, and one alternative per position scores identically to .phrase. This depends on tantivy internals — hence the =0.26.1 pin — and singleAlternativesScoreLikePhrase is the tripwire.
• Single-position path uses a DisjunctionMaxQuery of term queries wrapped in a ×1 BoostWeight: in tantivy 0.26.1 a top-level term disjunction collects top docs via block-WAND, which sums the alternatives regardless of the max combiner. Regression-tested with a document holding both alternatives.
• Cost caveat. Tantivy's regex-phrase weight allocates five segment-sized bitsets per position per segment. Negligible at app-local index sizes (~60KB/position at 100k docs); worth knowing at millions of docs.
• Fast fields on en_stem_keep hold both forms, so a terms aggregation returns designers and design as separate buckets. BM25 field length counts both tokens where they differ. Both noted on the enum case.
• Dependencies: rust-stemmers = "=1.2.0" and regex-syntax = "=0.8.10" become direct, pinned to tantivy's exact copies (cargo tree shows one copy of each). No new code ships.
• Not strictly additive at the source level: Analyzer and Query each gain a case, so exhaustive switches in callers need updating → minor version bump (0.8.0). Behaviour of existing analyzers, queries and indexes is unchanged; indexes using en_stem_keep or queries using multi_phrase need this library version.

Tests

392 Swift tests (36 new across AnalyzerTests, TokenizerTests, KeepStemPhraseTests, MultiPhraseTests, QueryEncodingTests) and 15 Rust tests pass. Build note for this machine: CARGO_PROFILE_RELEASE_BUILD_OVERRIDE_STRIP=false scripts/build-host.sh (cargo's release strip breaks proc-macro dylibs on rustc 1.96/macOS 27) and swift test --manifest-cache none after the host lib first appears.

🤖 Generated with Claude Code