Skip to content

Commit dd85ef4

Browse files
committed
docs: detail search improvements
Document the fork's query segmentation, ranking, normalization, and navigation changes with counterexamples and behavior boundaries. Assisted-by: OpenAI Codex (GPT-5)
1 parent 38c5c2e commit dd85ef4

1 file changed

Lines changed: 98 additions & 4 deletions

File tree

‎FORK.md‎

Lines changed: 98 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,9 @@ or a project explicitly enables an added extension or plugin.
88

99
## Included changes
1010

11-
- Chinese queries use complete, required prefix terms instead of ambiguous
12-
one-character OR matches.
13-
- Hyphenated technical terms require all components and prioritize complete
14-
matches, so queries such as `pre-commit` don't get dominated by `commit`.
11+
- [Search behavior](#search-behavior) is adjusted for Chinese segmentation,
12+
hyphenated technical terms, stemmed or punctuation-wrapped terms, exact
13+
phrase ranking, and post-navigation highlighting.
1514
- Code blocks receive unique copy targets when several blocks share an
1615
anchored ancestor.
1716
- `material.extensions.admonition` adds opt-in IDs and permalinks to
@@ -21,6 +20,101 @@ or a project explicitly enables an added extension or plugin.
2120
- Google Fonts stylesheets load asynchronously, so an unavailable font service
2221
doesn't block the initial render and the system fallback remains usable.
2322

23+
## Search behavior
24+
25+
These changes retain Lunr's normal field boosts, query syntax, prefix search,
26+
stemming, and stop-word handling. They correct specific cases where the query
27+
transformation, ranking, or navigation behavior produced a misleading result.
28+
29+
### Chinese query segmentation
30+
31+
A continuous run of Han characters is segmented from left to right using the
32+
longest term already present in the inverted index. Every resulting segment is
33+
required and retains a trailing wildcard for search-as-you-type. A complete
34+
indexed term is kept intact, the last character is no longer dropped, and an
35+
unmatched suffix is preserved as a prefix query.
36+
37+
Counterexample: given indexed terms `实时` and `调度`, the query `实时调度` used
38+
to become optional prefix terms. A document containing only the common
39+
one-character or partial term could therefore appear, while a segmentation bug
40+
could also omit the final character. The fork produces `+实时* +调度*`, so a
41+
result must match both parts. Given an indexed complete term `系统管理员`, the
42+
same query stays one term instead of being unnecessarily split. Given a
43+
partially typed `网络配`, the unmatched `配` is retained rather than discarded.
44+
45+
### Hyphenated technical terms
46+
47+
For simple hyphenated words, all searchable components must match. Documents
48+
containing the exact hyphenated spelling receive a bounded relevance boost.
49+
Stop words removed by the configured pipeline do not make the compound
50+
impossible to match. URLs, command-line options, prohibited terms, and other
51+
complex query syntax are not treated as compounds by this rule.
52+
53+
Counterexample: Material's separator splits `pre-commit` into the optional
54+
terms `pre` and `commit`. A section titled "Commit Message Convention" could
55+
then outrank the section that actually documents `pre-commit`, because the
56+
title boost is much larger than the body-text boost. The fork rejects results
57+
that match only `commit` and ranks literal `pre-commit` occurrences above
58+
documents that merely contain both words separately. This does not change the
59+
query `pre commit`, which intentionally remains an OR query.
60+
61+
### Stemmed and punctuation-wrapped terms
62+
63+
Query terms are passed through the configured Lunr search pipeline before an
64+
automatic wildcard is added. When stemming changes a term, the normalized term
65+
is searched without a wildcard, because Lunr disables its search pipeline for
66+
wildcard queries. During indexing, a token surrounded by punctuation also gets
67+
an alias without the leading and trailing punctuation.
68+
69+
Counterexample: the English stemmer normalizes `Relative` to `relat`. Searching
70+
for `Relative*` bypasses stemming and therefore cannot find the indexed
71+
`relat`. In prose such as `(Relative Distinguished Name,RDN)`, tokenizer
72+
boundaries can additionally leave `Relative` attached to full-width
73+
punctuation. The fork searches the normalized term and indexes an unwrapped
74+
alias, so this occurrence is discoverable. This alias only strips punctuation
75+
at token boundaries; it does not rewrite punctuation inside a term.
76+
77+
### Exact multi-term phrase ranking
78+
79+
A simple query containing two or more whitespace-separated words receives an
80+
additional bounded score when the original phrase occurs verbatim, ignoring
81+
case, collapsed whitespace, zero-width spaces, and search-result markup. The
82+
normal Lunr results are still used; this is a post-query ranking signal, not a
83+
phrase-search filter. Field selectors, modifiers, and other complex syntax are
84+
excluded from this extra boost.
85+
86+
Counterexample: for `x server`, Lunr's prefix matching and title boost can rank
87+
a heading named `XFS` first: `x*` matches the `xf` stem in that title, even
88+
though the section says nothing about an X server. The fork boosts the section
89+
whose text actually contains `X server`, placing it above the prefix-only title
90+
match. A query such as `x server` can still return documents containing the two
91+
terms separately; they simply do not receive the exact-phrase boost.
92+
93+
### Highlight terms and in-section navigation
94+
95+
When `search.highlight` is enabled, result URLs use the original query terms
96+
for highlighting instead of their stemmed index representation. After the
97+
marks are inserted, navigation starts at the result's section anchor and then
98+
scrolls the first highlighted occurrence at or after that anchor into the
99+
center of the viewport. Matches earlier on the same page are ignored for this
100+
scroll decision.
101+
102+
Counterexample: the English stemmer turns `XFS` into `xf`. The previous result
103+
URL was `?h=xf#xfs`, so only the `XF` substring was highlighted. The fork emits
104+
`?h=xfs#xfs`, which highlights the complete term. Separately, if a matching
105+
sentence is halfway through a long section, normal hash navigation stops at
106+
the heading and leaves the match off-screen; the fork continues to the first
107+
actual mark within or after that section. If the heading itself contains the
108+
term, its mark remains the first match and no unnecessary deeper jump occurs.
109+
110+
Enable this navigation behavior with the existing Material feature flag:
111+
112+
```yaml
113+
theme:
114+
features:
115+
- search.highlight
116+
```
117+
24118
## Installation
25119
26120
Until this fork has a separate package name and release channel, install a

0 commit comments

Comments
 (0)