Summary
On Japanese (lang="ja") sites, searching for a common compound noun such as 新幹線 ("Shinkansen" / bullet train) returns unrelated pages — ranked above the pages that actually contain the word — while searching for a fragment of it (幹線) returns exactly the right pages.
The root cause appears to be that the index and the query are segmented by different tokenizers that disagree on compounds:
- Index time (extended binary): 新幹線 is split into 新 + 幹線, so the index contains the words
新 and 幹線 but not 新幹線.
- Query time (
Intl.Segmenter with ICU's Japanese dictionary): 新幹線 is kept as a single word ([...new Intl.Segmenter('ja', {granularity:'word'}).segment('新幹線')] → one segment).
The query word 新幹線 therefore matches no indexed word, and the fallback appears to shorten the term from the end (新幹線 → 新幹 → 新) until something matches — landing on the extremely generic single-kanji word 新 ("new"), which exists on many pages via other splits (e.g. 新千歳空港 → 新 + 千歳 + 空港). Those unrelated pages then match, and can outrank the pages that genuinely contain 新幹線. The discriminative part of the query (幹線) — which is in the index — is exactly the part that gets discarded.
The v1.5.0 release notes describe the query-side Intl.Segmenter as segmenting queries the same way the content was indexed, but for Japanese the two sides use different dictionaries (Lindera vs ICU), so they frequently disagree on compounds.
Reproduction
site/shinkansen.html — the page a user searching 新幹線 wants:
<!doctype html>
<html lang="ja">
<head><title>東京から大阪への移動手段</title></head>
<body data-pagefind-body><h1>東京から大阪への移動手段</h1><p>東京から大阪への移動は新幹線が便利です。新幹線の切符は駅の窓口で購入できます。</p></body>
</html>
site/chitose.html — an unrelated page (about 新千歳空港 airport, no mention of 新幹線):
<!doctype html>
<html lang="ja">
<head><title>札幌の空港アクセス</title></head>
<body data-pagefind-body><h1>札幌の空港アクセス</h1><p>新千歳空港から札幌駅まではバスで移動できます。新千歳空港は北海道の玄関口です。</p></body>
</html>
site/search.html — a minimal Default UI page:
<!doctype html>
<html lang="ja">
<head>
<title>検索</title>
<link rel="stylesheet" href="/pagefind/pagefind-ui.css" />
<script src="/pagefind/pagefind-ui.js"></script>
</head>
<body>
<div id="search"></div>
<script>
window.addEventListener('DOMContentLoaded', () => {
new PagefindUI({ element: '#search' });
});
</script>
</body>
</html>npx pagefind@1.5.2 --site site --serve
# open http://localhost:1414/search.html
| Query |
Expected |
Actual (1.5.2, Chromium) |
新幹線 |
only 東京から大阪への移動手段 |
札幌の空港アクセス ranked first, then 東京から大阪への移動手段 |
幹線 |
東京から大阪への移動手段 |
東京から大阪への移動手段 ✅ |
So a fragment of the compound finds the right page, while the full, natural query surfaces an unrelated page above it. (Via the JS API the scores are 0.28 for the unrelated page vs 0.26 for the correct one.)
Decoding the generated pagefind/index/*.pf_index chunks confirms the vocabulary: the index contains 新, 幹線, 千歳, 空港, … but no 新幹線.
Real-world impact
On our 189-page Japanese blog, the query 新幹線 returns 33 results led by pages about 新千歳空港 and 新宿 (which merely contain the word 新 after segmentation); the handful of pages that actually discuss 新幹線 are only findable by typing 幹線. An exact-phrase query "新幹線" returns 0 results, since no indexed word can match. Any Japanese compound that ICU keeps whole but Lindera splits behaves this way; the failure is invisible when the leftover prefix happens to be discriminative (駐車場 → 駐車 works fine) and severe when it is generic (新幹線 → 新).
Environment
- Pagefind 1.5.2 (extended, via
npx pagefind)
- Observed in Chromium 140 (Playwright) and reproduced in Node.js 22 (both use ICU for
Intl.Segmenter)
- The production site is a static Astro 6 build with Pagefind run directly via the CLI, but Astro is not involved — the repro above uses plain HTML files and
npx pagefind only
Workaround we're shipping
Edit 2: earlier versions of this section rewrote the query by AND-ing the compound's leading/trailing bigrams onto it (新幹線 → 新幹線 新幹 幹線), first via processTerm, then with a zero-result fallback. Both lost recall, and measuring why surfaced a fact that makes any query-side rewrite harder than it looks: Lindera segments the same compound differently depending on context, within the same index. In our index 国内線 ("domestic flights") is sometimes 国内+線 and sometimes 国+内線, so requiring both bigrams drops every page that only contains one of the splits (国内線: 70 → 50 hits, 最寄駅: 10 → 2). Meanwhile compounds that Lindera keeps whole (日暮里) don't contain their internal bigrams at all, which zeroed the query entirely. The version below no longer replaces the result set — it only reorders it.
Current approach, per kanji-only query word of 3+ characters:
- Probe with a quoted exact search (
"日暮里") whether the compound exists verbatim as an indexed word. If it does, the plain query already behaves well — leave it untouched.
- If it doesn't (新幹線, 国内線), run both the bigram-expanded query and the plain query, and return the union: expanded results first, then the plain query's remaining results deduped. Recall is now always identical to plain Pagefind; the expansion only promotes the pages that contain the compound's fragments (for 新幹線, the 7 genuine pages now rank above the 40 新* prefix matches).
Since this needs result counts and merging, it can't live in processTerm. We wrap the Pagefind API module and point the Default UI at the wrapper via its bundlePath option. The wrapper is a static file served at /pagefind-mod/pagefind.js alongside the real bundle (in an Astro project: public/pagefind-mod/pagefind.js; the real bundle stays at /pagefind/ — the wrapper imports it by absolute URL, and since the real module resolves its assets from its own import.meta.url, the index still loads from /pagefind/):
// /pagefind-mod/pagefind.js
// Only search(), debouncedSearch() and preload() are wrapped; everything else
// (createInstance, mergeIndex, …) is re-exported from the real bundle as-is.
export * from '/pagefind/pagefind.js';
import {
search as baseSearch,
preload as basePreload,
} from '/pagefind/pagefind.js';
const segmenter =
typeof Intl !== 'undefined' && typeof Intl.Segmenter !== 'undefined'
? new Intl.Segmenter('ja', { granularity: 'word' })
: null;
// Skip queries we can't or shouldn't rewrite: no Intl.Segmenter, no kanji, or
// quote characters anywhere (rebuilding the term would break phrase syntax).
function isRewritable(term) {
return (
segmenter !== null &&
typeof term === 'string' &&
!term.includes('"') &&
/\p{Script=Han}/u.test(term)
);
}
function segmentWords(term) {
const words = [];
for (const { segment } of segmenter.segment(term)) {
const word = segment.trim();
if (word) words.push(word);
}
return words;
}
function isExpandableWord(word) {
return Array.from(word).length >= 3 && /^\p{Script=Han}+$/u.test(word);
}
function bigrams(word) {
const chars = Array.from(word); // code points, not UTF-16 units
return [chars.slice(0, 2).join(''), chars.slice(-2).join('')];
}
function expandedQuery(words, expandWord) {
return words
.flatMap((w) => (expandWord(w) ? [w, ...bigrams(w)] : [w]))
.join(' ');
}
export async function search(term, options) {
if (!isRewritable(term)) return baseSearch(term, options);
const words = segmentWords(term);
const compounds = words.filter(isExpandableWord);
if (compounds.length === 0) return baseSearch(term, options);
const probes = await Promise.all(compounds.map((w) => baseSearch(`"${w}"`)));
const missing = new Set(
compounds.filter((_, i) => probes[i].results.length === 0),
);
if (missing.size === 0) return baseSearch(term, options);
const [expanded, plain] = await Promise.all([
baseSearch(
expandedQuery(words, (w) => missing.has(w)),
options,
),
baseSearch(term, options),
]);
if (expanded.results.length === 0) return plain;
const promoted = new Set(expanded.results.map((r) => r.id));
return {
...plain,
results: [
...expanded.results,
...plain.results.filter((r) => !promoted.has(r.id)),
],
};
}
let lastDebounceId = 0;
export async function debouncedSearch(term, options, debounceTimeoutMs = 300) {
const id = ++lastDebounceId;
await new Promise((resolve) => setTimeout(resolve, debounceTimeoutMs));
if (id !== lastDebounceId) return null; // superseded by a newer call
return search(term, options);
}
export async function preload(term, options) {
const tasks = [basePreload(term, options)];
if (isRewritable(term)) {
const expanded = expandedQuery(segmentWords(term), isExpandableWord);
if (expanded !== term) tasks.push(basePreload(expanded, options));
}
await Promise.all(tasks);
}new PagefindUI({ element: '#search', bundlePath: '/pagefind-mod/' });Regression-tested against 20 common queries on our 189-page index: every query now returns exactly the same result set as unpatched Pagefind (国内線 70/70, 最寄駅 10/10, 国際線 98/98, 日暮里 19/19 in identical order), with only the ordering improved where the mismatch bites (新幹線's 7 genuine pages occupy the top 7 of 47). Still clearly a band-aid — and the context-dependent segmentation above suggests a query-side heuristic can never fully reconstruct what the indexer did.
Possible directions
A few ideas, in case they're useful:
- Segment queries against the index's own vocabulary (longest-match over indexed words) instead of — or as a fallback after —
Intl.Segmenter, so the query can only produce words that can actually match.
- When shortening an unmatched CJK term, keep searching the remainder rather than discarding it: 新幹線 → 新 + 幹線 instead of just 新. The discarded tail is often the discriminative part.
- Emit ICU-style segmentation at index time as well (e.g. via
icu_segmenter), so both vocabularies coexist in the index. ICU4X and browser ICU4C dictionaries still differ slightly, but far less than Lindera vs ICU.
Happy to provide the full repro as a tarball or test against a branch. Thanks for Pagefind — aside from this, it's been great to work with on a bilingual (ja/en) static blog.
The investigation (index decoding, segmenter comparison, minimal repro) and this write-up were done together with Claude (Anthropic's Fable 5); the recall regression in the earlier workaround was caught by a code review from OpenAI's GPT-5.6 (Sol). Everything was reviewed and reproduced by a human before filing.
Summary
On Japanese (
lang="ja") sites, searching for a common compound noun such as 新幹線 ("Shinkansen" / bullet train) returns unrelated pages — ranked above the pages that actually contain the word — while searching for a fragment of it (幹線) returns exactly the right pages.The root cause appears to be that the index and the query are segmented by different tokenizers that disagree on compounds:
新and幹線but not新幹線.Intl.Segmenterwith ICU's Japanese dictionary): 新幹線 is kept as a single word ([...new Intl.Segmenter('ja', {granularity:'word'}).segment('新幹線')]→ one segment).The query word 新幹線 therefore matches no indexed word, and the fallback appears to shorten the term from the end (新幹線 → 新幹 → 新) until something matches — landing on the extremely generic single-kanji word
新("new"), which exists on many pages via other splits (e.g. 新千歳空港 → 新 + 千歳 + 空港). Those unrelated pages then match, and can outrank the pages that genuinely contain 新幹線. The discriminative part of the query (幹線) — which is in the index — is exactly the part that gets discarded.The v1.5.0 release notes describe the query-side
Intl.Segmenteras segmenting queries the same way the content was indexed, but for Japanese the two sides use different dictionaries (Lindera vs ICU), so they frequently disagree on compounds.Reproduction
site/shinkansen.html— the page a user searching 新幹線 wants:site/chitose.html— an unrelated page (about 新千歳空港 airport, no mention of 新幹線):site/search.html— a minimal Default UI page:npx pagefind@1.5.2 --site site --serve # open http://localhost:1414/search.html新幹線幹線So a fragment of the compound finds the right page, while the full, natural query surfaces an unrelated page above it. (Via the JS API the scores are 0.28 for the unrelated page vs 0.26 for the correct one.)
Decoding the generated
pagefind/index/*.pf_indexchunks confirms the vocabulary: the index contains新,幹線,千歳,空港, … but no新幹線.Real-world impact
On our 189-page Japanese blog, the query 新幹線 returns 33 results led by pages about 新千歳空港 and 新宿 (which merely contain the word
新after segmentation); the handful of pages that actually discuss 新幹線 are only findable by typing 幹線. An exact-phrase query"新幹線"returns 0 results, since no indexed word can match. Any Japanese compound that ICU keeps whole but Lindera splits behaves this way; the failure is invisible when the leftover prefix happens to be discriminative (駐車場 → 駐車 works fine) and severe when it is generic (新幹線 → 新).Environment
npx pagefind)Intl.Segmenter)npx pagefindonlyWorkaround we're shipping
Current approach, per kanji-only query word of 3+ characters:
"日暮里") whether the compound exists verbatim as an indexed word. If it does, the plain query already behaves well — leave it untouched.Since this needs result counts and merging, it can't live in
processTerm. We wrap the Pagefind API module and point the Default UI at the wrapper via itsbundlePathoption. The wrapper is a static file served at/pagefind-mod/pagefind.jsalongside the real bundle (in an Astro project:public/pagefind-mod/pagefind.js; the real bundle stays at/pagefind/— the wrapper imports it by absolute URL, and since the real module resolves its assets from its ownimport.meta.url, the index still loads from/pagefind/):Regression-tested against 20 common queries on our 189-page index: every query now returns exactly the same result set as unpatched Pagefind (国内線 70/70, 最寄駅 10/10, 国際線 98/98, 日暮里 19/19 in identical order), with only the ordering improved where the mismatch bites (新幹線's 7 genuine pages occupy the top 7 of 47). Still clearly a band-aid — and the context-dependent segmentation above suggests a query-side heuristic can never fully reconstruct what the indexer did.
Possible directions
A few ideas, in case they're useful:
Intl.Segmenter, so the query can only produce words that can actually match.icu_segmenter), so both vocabularies coexist in the index. ICU4X and browser ICU4C dictionaries still differ slightly, but far less than Lindera vs ICU.Happy to provide the full repro as a tarball or test against a branch. Thanks for Pagefind — aside from this, it's been great to work with on a bilingual (ja/en) static blog.
The investigation (index decoding, segmenter comparison, minimal repro) and this write-up were done together with Claude (Anthropic's Fable 5); the recall regression in the earlier workaround was caught by a code review from OpenAI's GPT-5.6 (Sol). Everything was reviewed and reproduced by a human before filing.