Skip to content

README.md

Latest commit

 

History

History
90 lines (71 loc) · 3.4 KB

README.md

File metadata and controls

90 lines (71 loc) · 3.4 KB
Raw

Baseline Reference Runners

This directory provides compact reference runners for five baselines used in the paper. The default output budget is 10,000 records.

Directory Packaged component Paper-aligned behavior
random/ project-authored runner uniform sampling without replacement, seed 42
superfiltering/ project-authored runner IFD ratio, threshold below 1, descending rank
selectit/ project-authored clean-room runner token-, sentence-, and model-level uncertainty-aware self-reflection
dsir/ thin runner plus official DSIR core hashed unigrams and bigrams in 10,000 buckets, importance resampling without replacement
tsds/ thin runner plus official TSDS core task/query and candidate embeddings, KNN-KDE/OT assignment, multinomial sampling with replacement

The DSIR and TSDS source under each vendor/ directory is third-party code and retains its MIT license. The surrounding runners only normalize the repository's JSON/JSONL SFT schema and expose the paper protocol as command-line arguments.

Install the baseline-specific dependencies after creating the main environment:

pip install -r baselines/requirements.txt
pip install -e baselines/dsir/vendor

Input schema

Random, SuperFiltering, SelectIT, and DSIR accept a JSON array or JSONL file. Each record should contain instruction, optional input, and output fields. TSDS accepts the same candidate file plus two NumPy embedding matrices. Candidate embeddings must follow candidate-file order.

Commands

Run commands from the repository root.

python baselines/random/select_random.py \
  --input /path/to/pool.json \
  --output /path/to/random-10k.json
python baselines/superfiltering/select_superfiltering.py \
  --input /path/to/pool.json \
  --output /path/to/superfiltering-10k.json \
  --model /path/to/gpt2-style-proxy

SelectIT requires a rating-prompt file and one or more foundation models. Its default model level also requires the corresponding parameter counts:

python baselines/selectit/run_selectit.py \
  --input /path/to/pool.json \
  --output /path/to/selectit-10k.json \
  --rating-prompts /path/to/rating_prompt.txt \
  --models /path/to/model-7b /path/to/model-13b /path/to/model-70b \
  --parameter-counts 7 13 70

See selectit/README.md for the clean-room implementation boundary and the external assets required for exact run reconstruction.

python baselines/dsir/run_dsir.py \
  --input /path/to/pool.json \
  --target /path/to/target-examples.json \
  --output /path/to/dsir-10k.json
python baselines/tsds/run_tsds.py \
  --input /path/to/pool.json \
  --candidate-embeddings /path/to/candidate.npy \
  --query-embeddings /path/to/query.npy \
  --output /path/to/tsds-10k.json \
  --indices-output /path/to/tsds-indices.npy

TSDS follows the upstream method's sampling-with-replacement semantics. Its materialized JSON output may consequently contain duplicate records. The runner does not oversample, deduplicate, or rerank the official output.

Scope

These runners expose the selection procedures and expected data interfaces. The repository does not include full source pools, target/reference sets, embeddings, model weights, baseline-selected subsets, or baseline result tables. Exact comparison-table reproduction additionally requires the paper's pinned external assets, SFT settings, and evaluation benchmarks.