[ilyazub]

AI-generated: Claude Opus 5.5, high effort: code review, tests and benchmarks (wrote commits 1beeba5, fc7ad2d and d8242a9 and this text; the earlier commits came from earlier sessions of the same work and were reviewed here).

Repeated css/at_css calls on a large document now look up tag, class, ID and attribute candidates in an index the document keeps until its next DOM write, instead of walking the whole tree each time. Results and their order are unchanged. Stacked on #48 (its two commits are the first ones here): merge that first, then this.

Why. SerpApi's Google parser makes about 2,000 css/at_css calls per 0.5-1.5 MB page, most of them from the root, and each call walked every node.

Effect, Nokolexbor alone (d8242a9). Instructions retired (Apple thread_selfcounts) for 8 real Google pages, each parsed once and then queried with 300 selectors from SerpApi's parsers via css and at_css; median of 5 interleaved runs per arm [min-max]:

Ruby	master 0375a7b	this PR	change
2.7.8	1,481 M [1,478-1,485]	296 M [295-296]	−80 %
3.3.10	1,481 M [1,480-1,487]	301 M [300-301]	−80 %
3.3.10 YJIT	1,482 M [1,476-1,484]	298 M [297-298]	−80 %
3.4.8	1,485 M [1,479-1,486]	299 M [299-303]	−80 %
4.0.7	1,470 M [1,463-1,471]	285 M [284-286]	−81 %
parsing the same pages (control, 2.7.8)	102.7 M [101.9-103.8]	102.8 M [102.4-104.1]	within noise

Effect on a whole parser (measured on fc7ad2d, before the variable reads were removed). SerpApi's Search#html_to_hash over 24 Google pages (one per parser feature), instructions per pass, median of 5 interleaved rounds [min-max]:

Ruby	master 0375a7b	this PR	change
2.7.8	29,681 M [29,620-29,821]	23,903 M [23,885-24,002]	−19.5 %
3.3.10	34,767 M [34,422-35,739]	28,881 M [28,695-29,733]	−16.9 %
3.3.10 YJIT	36,289 M [36,046-37,142]	30,609 M [30,388-31,360]	−15.7 %

Ruby 4.0 could not run SerpApi (its lockfile pins commonmarker 0.23.12, which requires Ruby < 4.0); 3.4.8 ran with all four open Nokolexbor PRs together: 33,648 M → 24,699 M (−26.6 %). The YJIT arm had one warm-up pass, so some compilation is counted in both arms. On 20 pages from 10 other engines the median moved +2.5 to +3.1 % with overlapping ranges; Google Shopping AI-overview pages (2.5-4 billion instructions, GC-heavy) went +6.7 % on 2.7.8 with separate ranges but −15 % and +10 % with overlapping ranges on 3.3.10 and YJIT, so single-iteration rounds can't settle them.

Memory. The index lives as long as the document: +0.1 to +0.3 MB per document after 300 searches on the same 8 pages (documents of 0.8-4.5 MB), jemalloc stats.allocated, Ruby 2.7.8.

Output. SerpApi's Google parser over the 2,760 searches its specs parse, JSON compared byte for byte against master 0375a7b: 0 differ (Ruby 2.7.8, d8242a9); with all four open Nokolexbor PRs combined (as of fc7ad2d), also 0 differ on Ruby 3.3.10, 3.3.10 YJIT and 3.4.8. All-engine corpus (7,086 searches, 124 engines) with this PR combined with the other three open Nokolexbor PRs: 41 differ raw, 0 after normalizing per-run search IDs in links and object addresses in error messages (master 0375a7b vs the combined build, Ruby 2.7.8).

How it works:

• 0007-lexbor-index-descendant-candidates.patch: the index lists the tree in pre-order with each node's subtree end, plus, per tag, class, ID and attribute name, the sorted positions of the nodes that carry it. When Lexbor's matcher looks for the next descendant matching a compound that names one of those, it binary-searches that key's positions inside the subtree and tests only those candidates, in document order, instead of walking every node.
• 0008-lexbor-cache-selectors-by-document.patch: the index for a whole document is built on the first search from the root and kept on lxb_dom_document_t with a version number. Every native DOM writer in the binding (add_child/add_sibling, remove, destroy, content=, []=, remove_attr, Attribute#name=/#value=, title=, fragment parsing, and NodeSet#css's temporary reparenting) calls lxb_selectors_index_invalidate. Searches from any node inside an indexed document reuse its index; large detached trees can build a temporary per-call index.
• Matching holds the GVL, so two threads searching the same document cannot interleave on its index.
• extconf.rb now applies the patches sorted by name: Ruby 2.7's Dir[] is unsorted, and patch 0008 edits lines 0007 adds.

No configuration: there is no switch and nothing reads the environment. A build compiled with NOKOLEXBOR_SELECTOR_CHECK=1 (also implied by NOKOLEXBOR_ASAN=1, so CI's ASAN job gets it) runs Lexbor's plain traversal next to every css/at_css call and aborts on any difference in matched nodes or their order; release builds don't contain that code. The specs assert literal results derived from how each test document is built (they pass unchanged on master, which has no index), so they run in the default build without variables.

Cost of carrying the Lexbor patches. The index is two patches on vendored Lexbor v2.1.0-55: 0007 (646 lines) and 0008 (421 lines), touching selectors/selectors.c, selectors/selectors.h and dom/interfaces/document.h. Every Lexbor upgrade has to re-port them, including a move to Lexbor v3 (the closed port #42 lists the API changes there); they hook into lxb_selectors_find and the matcher's descendant search. Offering them upstream to Lexbor would remove that cost; not attempted here.

Related issues. #46 (open, jvmvik is working on it) proposes memoizing whole css/at_css results per document, invalidated by a mutation epoch bumped in every binding writer. #45 proposes caching parsed selector lists per thread. This PR makes each traversal cheaper instead: it stores no results, so it needs no result keys or size bounds, every selector reuses the same per-document index, and selector parsing is unchanged. #46 needs exactly the writer hooks this PR adds (lxb_selectors_index_invalidate in every native DOM writer); if both land, one invalidation point should serve both. #45 is orthogonal and would stack with this.

Not covered: Windows and the precompiled (rake-compiler-dock) gems were not built; Ruby 3.1 and 3.2 not run. The all-engine corpus and the whole-parser instruction counts ran before d8242a9 (which only removes the variable reads and the unbuilt check code); the Nokolexbor-only counts, Google gate, specs and sanitizers ran on d8242a9.

Checks on d8242a9 (macOS arm64 unless noted, base master 0375a7b)

Clean git archive, gem build, gem install, specs against the installed gem with lib/ off the load path (CI's test:gem):

Ruby 2.7.8   319 runs, 4705 assertions, 0 failures, 0 errors, 0 skips
Ruby 3.3.10  319 runs, 4692 assertions, 0 failures, 5 errors   (nokogiri not installed; same 5 on master)
Ruby 3.4.8   319 runs, 4705 assertions, 0 failures, 0 errors, 0 skips
Ruby 4.0.7   319 runs, 4692 assertions, 0 failures, 5 errors   (same)
Ruby 2.7.8, NOKOLEXBOR_SELECTOR_CHECK=1 build: 320 runs, 4709 assertions, 0 failures (adds the index-reuse test)
spec/selector_index_spec.rb against master 0375a7b: 6 runs, 40 assertions, 0 failures

Patches: all eight apply in name order to a clean Lexbor b2c0a61, and the result equals the tree the Ruby 2.7.8 gem install built.

ASAN + UBSAN, Debian 12 arm64, gcc 12, Ruby 3.1 (lexbor and the extension instrumented; NOKOLEXBOR_ASAN=1 turns the selector check on, so every search below was also compared with the plain traversal):

spec suite                    320 runs, 0 failures, 0 skips, 0 reports (heap checks; see note)
16-thread stress              7,680 css/at_css calls, 8 Google pages, 3,277 SerpApi selectors,
                              nested NodeSet searches, add_child/[]=/content=/remove, GC.start, GC.stress spots: 0 errors, 0 reports
mutation fuzzer               100,001 operations, 39,343 indexed-vs-plain comparisons, 842 NodeSet, 418 overlapping-set,
                              21,315 DOM edits across 11 writers, 45 GC.stress operations: 0 mismatches (the check aborts on the first), 0 reports
selector_index_recovery_test  pass, 0 reports
TSan (CI's test/tsan_race_test.c, macOS clang)   0 races (master 0)

Note: with stack instrumentation on, the suite stops at one stack-buffer-overflow on a store to a local struct in nl_node_find. Debian's Ruby unwinds exceptions with __builtin_longjmp, which ASAN does not intercept, so earlier raises leave stale redzones on the stack. The run above uses --param=asan-stack=0 (heap, global and UBSAN checks stay on); master's suite with stack instrumentation reported nothing.

CI note: the Windows install-gem-and-test jobs fail on master 0375a7b too (cannot load such file -- nokolexbor, CI run of 2026-09-27), so they are not caused by this change. Changed C lines pass the repository's .clang-format (git clang-format --diff origin/master, clang-format 23.1.3).