Skip to content

Implement exponential search for rank -> docid batch lookup#2937

Open
waamm wants to merge 2 commits into
quickwit-oss:mainfrom
waamm:exponential-search
Open

Implement exponential search for rank -> docid batch lookup#2937
waamm wants to merge 2 commits into
quickwit-oss:mainfrom
waamm:exponential-search

Conversation

@waamm

@waamm waamm commented May 19, 2026

Copy link
Copy Markdown

This PR addresses an almost 2 year old TODO by @PSeitz: instead of scanning doc-by-doc to find which document owns a given value rank, we now bracket the answer exponentially and finish with a monotone binary search. Quick informal timings suggest this is roughly ~5× faster on some workloads.

Tests include cases against the old linear-scan implementation.

Edit: On a 10M-document multivalued column where every doc has 2 values, a single select_batch_in_place call converting 10k evenly spaced row ranks (on an M4 Max):

10M nonempty / 10k ranks exponential
select_batch_single_batch        Avg: 2.2253ms (-3.87%)    Median: 2.2268ms (-3.83%)    [2.1889ms .. 2.2818ms]    
10M nonempty / 10k ranks linear
select_batch_single_batch        Avg: 11.9561ms (-3.13%)    Median: 11.8640ms (-3.45%)    [11.7694ms .. 14.0043ms] 

@fulmicoton

Copy link
Copy Markdown
Collaborator

Are there benchmark confirming this is improving something? Can you paste the results in the PR description?

@waamm

waamm commented Jun 8, 2026

Copy link
Copy Markdown
Author

Hi! I added a description of the benchmark I had in mind (I can also add the code).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants