Repository navigation
Add code-benchmarks replication kit and deprecate the unused cai_extract.py - #61
Merged
Merged
Conversation
amin-rigi
previously approved these changes
Oct 7, 2026
…act.py code-benchmarks/ is a self-contained, offline replication of the .cai vs Graphify vs raw-files code-retrieval numbers. run_all.py rebuilds every .cai store from source with the tree-sitter extractor and regenerates results/RESULTS.md. The summary tables carry .cai-vs-Graphify and .cai-vs-raw token-reduction columns. The requests most-imported tie is graded any-of, and two tautological per-language questions plus a verbosity-penalising grader were corrected. Kerneta-Cai/engine/cai_extract.py becomes a deprecation stub: it is the legacy AST-only extractor nothing in the product uses, and building a store with it instead of cai_ts_extract.py silently drops extends edges and under-captures imports. The stub errors with a pointer, and Kerneta-Cai/BENCHMARK-REPRODUCE.md documents the correct extractor. keyless_check skips the code-benchmarks directory, whose ES-module .js fixtures are not product source and do not parse as CommonJS.
Kerneta
force-pushed
the
bench/code-benchmarks-and-extractor-guardrail
branch
from
October 7, 2026 05:11
c32f0bf to
5904f71
Compare
amin-rigi
self-requested a review
October 7, 2026 05:15
amin-rigi
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
code-benchmarks/ is a self-contained, offline replication of the .cai vs
Graphify vs raw-files code-retrieval numbers.
python code-benchmarks/run_all.pyrebuilds every .cai store from source with the tree-sitter extractor and
regenerates
results/RESULTS.md. No API key and no network: structural toolsparse offline and grading is programmatic, so anyone can rebuild every number.
The summary tables now carry two token-reduction columns,
.cai vs Graphifyand
.cai vs raw, expressed in x (for example 164x less).Reproduced from source:
Benchmark fairness fixes (question sets, not the engine)
modelsandcompattie at indegree 10.questions whose gold equalled the query argument, and converted two set
questions to presence so verbose baselines are not penalised for being
correct but wordy. .cai still scores full marks.
Engine change
Kerneta-Cai/engine/cai_extract.pybecomes a deprecation stub. It is the legacyAST-only extractor that nothing in the product uses; building a store with it
instead of
cai_ts_extract.pysilently drops extends edges and under-capturesimports. The stub errors with a pointer to the right extractor, and
Kerneta-Cai/BENCHMARK-REPRODUCE.mddocuments it so no one repeats the mistake.Verify