Skip to content

Add code-benchmarks replication kit and deprecate the unused cai_extract.py - #61

Merged
amin-rigi merged 1 commit into
mainfrom
bench/code-benchmarks-and-extractor-guardrail
Oct 7, 2026
Merged

amin-rigi merged 1 commit into
mainfrom
bench/code-benchmarks-and-extractor-guardrail

Conversation

@Kerneta

@Kerneta Kerneta commented Oct 7, 2026

Copy link
Copy Markdown
Owner

What this adds

code-benchmarks/ is a self-contained, offline replication of the .cai vs
Graphify vs raw-files code-retrieval numbers. python code-benchmarks/run_all.py
rebuilds every .cai store from source with the tree-sitter extractor and
regenerates results/RESULTS.md. No API key and no network: structural tools
parse offline and grading is programmatic, so anyone can rebuild every number.

The summary tables now carry two token-reduction columns, .cai vs Graphify
and .cai vs raw, expressed in x (for example 164x less).

Reproduced from source:

  • psf/requests (ast oracle): .cai 16/16 at 55 tok, 13x less than Graphify, 164x less than raw
  • Real-code retrieval: 10/10 at 24 tok, 5x / 202x less
  • Deep multi-hop: 11/11 at 27 tok, 4x / 7x less
  • Multi-language (python, js, ts, ruby): 44/44 at 33 tok, 4x / 8x less
  • httpx structural, all .cai at 100% recall: imports 218x less, subclasses 73x, tdeps 21x, callers 13x, callees 8x

Benchmark fairness fixes (question sets, not the engine)

  • The requests most-imported question is graded any-of, since models and
    compat tie at indegree 10.
  • The per-language synthetic set dropped one empty-type question, corrected two
    questions whose gold equalled the query argument, and converted two set
    questions to presence so verbose baselines are not penalised for being
    correct but wordy. .cai still scores full marks.

Engine change

Kerneta-Cai/engine/cai_extract.py becomes a deprecation stub. It is the legacy
AST-only extractor that nothing in the product uses; building a store with it
instead of cai_ts_extract.py silently drops extends edges and under-captures
imports. The stub errors with a pointer to the right extractor, and
Kerneta-Cai/BENCHMARK-REPRODUCE.md documents it so no one repeats the mistake.

Verify

cd code-benchmarks
pip install -r requirements.txt   # tree-sitter grammars
python run_all.py                 # regenerates results/RESULTS.md

@Kerneta
Kerneta requested a review from amin-rigi October 7, 2026 05:05
amin-rigi
amin-rigi previously approved these changes Oct 7, 2026
…act.py

code-benchmarks/ is a self-contained, offline replication of the .cai vs
Graphify vs raw-files code-retrieval numbers. run_all.py rebuilds every .cai
store from source with the tree-sitter extractor and regenerates
results/RESULTS.md. The summary tables carry .cai-vs-Graphify and .cai-vs-raw
token-reduction columns. The requests most-imported tie is graded any-of, and
two tautological per-language questions plus a verbosity-penalising grader were
corrected.

Kerneta-Cai/engine/cai_extract.py becomes a deprecation stub: it is the legacy
AST-only extractor nothing in the product uses, and building a store with it
instead of cai_ts_extract.py silently drops extends edges and under-captures
imports. The stub errors with a pointer, and Kerneta-Cai/BENCHMARK-REPRODUCE.md
documents the correct extractor.

keyless_check skips the code-benchmarks directory, whose ES-module .js fixtures
are not product source and do not parse as CommonJS.
@Kerneta
Kerneta force-pushed the bench/code-benchmarks-and-extractor-guardrail branch from c32f0bf to 5904f71 Compare October 7, 2026 05:11
@amin-rigi
amin-rigi self-requested a review October 7, 2026 05:15
@amin-rigi
amin-rigi merged commit 1d50a18 into main Oct 7, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants