|
| 1 | +# RSI research harness |
| 2 | + |
| 3 | +Research tooling only. Model-driven suites incur provider charges; the self-tests below do not. |
| 4 | +The learning loops, v1 delivery, bootstrap, and drift suites require the **`learn` features from |
| 5 | +PR #1405**. This harness-only branch does not supply them. Use a checkout containing those |
| 6 | +features through `ALTIMATE_CMD`; changing product code is outside this experiment. |
| 7 | +The verifier, fake backend, and local `none`/`gold`/`playbook` evaluation arms do not need `learn`. |
| 8 | + |
| 9 | +Prerequisites: Bun, Git, repository dependencies installed for the selected CLI, and Python |
| 10 | +3.10+ for the dbt environment (harness syntax supports 3.9+). The local checks used Python 3.11. |
| 11 | +For example, from the repository root: |
| 12 | + |
| 13 | +```bash |
| 14 | +python3.11 -m venv /tmp/rsi-dbt |
| 15 | +/tmp/rsi-dbt/bin/pip install dbt-core==1.12.5 dbt-duckdb==1.11.0 |
| 16 | +export DBT_BIN=/tmp/rsi-dbt/bin/dbt |
| 17 | +export DBT_PYTHON=/tmp/rsi-dbt/bin/python |
| 18 | +``` |
| 19 | + |
| 20 | +Pin package versions and record the CLI commit and model IDs alongside any reported results. |
| 21 | +The shared default CLI is this repository's `packages/opencode/src/index.ts`; baseline and fix |
| 22 | +comparison wrappers retain their historical `rsi-base` and `rsi` worktrees. An override is a |
| 23 | +shell-quoted command string, e.g. `ALTIMATE_CMD="bun run --conditions=browser '/path with spaces/src/index.ts'"`. |
| 24 | +Model-driven runs need provider authentication usable from the isolated homes. Account/org policies |
| 25 | +may block Claude, including Claude on Vertex; choose models your account can use. The harness checks |
| 26 | +requested model IDs before evaluation and treats timeouts, errors and incomplete turns as failed runs. |
| 27 | + |
| 28 | +## Running suites |
| 29 | + |
| 30 | +Free local checks (from the repository root): |
| 31 | + |
| 32 | +```bash |
| 33 | +experiments/rsi-workspace/demo/run_task.sh selftest |
| 34 | +python3 experiments/rsi-workspace/fake-backend/selftest.py |
| 35 | +python3 experiments/rsi-workspace/harness/selftest.py |
| 36 | +python3 experiments/rsi-workspace/harness/v1bench/selftest.py |
| 37 | +``` |
| 38 | + |
| 39 | +Model-driven suites, from `experiments/rsi-workspace/harness/`: |
| 40 | + |
| 41 | +```bash |
| 42 | +bash run_all.sh my-ci-loop # train / validation gate / publish / heldout arms |
| 43 | +bash run_corrections.sh my-corrections # simulated teammate corrections / publish / eval |
| 44 | +python3 eval.py --arm gold --runs 1 --run-id my-eval --out runs/my-eval/eval/gold.jsonl |
| 45 | +bash run_arms.sh my-ci-loop # rerun final arms of an existing completed loop |
| 46 | +python3 publish_replace.py runs/my-ci-loop # recover publishing using supported publish/update |
| 47 | +bash budget/run_budget.sh # rule-budget arms |
| 48 | +bash budget/run_budget2.sh # scope/conflict arms |
| 49 | +bash budget/run_drift.sh my-drift "${REFLECTOR_MODEL:-google-vertex/gemini-3.5-flash}" |
| 50 | +FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix.sh |
| 51 | +FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix2.sh |
| 52 | +bash v1bench/run_baselines.sh # n50-short / all-50 / all-300 / all-1000 |
| 53 | +bash v1bench/run_all_v1.sh # retrieval, vague task, topic, bootstrap, drift suites |
| 54 | +bash v1bench/run_fix_v1.sh # focused fix comparison (ALTIMATE_CMD selects checkout) |
| 55 | +``` |
| 56 | + |
| 57 | +See [v1 suite plan](harness/v1bench/PLAN.md), [topic switch](harness/v1bench/topic_switch/README.md), |
| 58 | +[vague tasks](harness/v1bench/vague_tasks/README.md), [demo](demo/README.md), and |
| 59 | +[fake backend](fake-backend/README.md) for task definitions and additional switches. |
| 60 | +`fake-backend/demo.sh` runs the model-driven two-user workspace demo. |
| 61 | +IDs must be simple names (letters/digits plus `.`, `_`, `-`); empty, absolute and traversing IDs |
| 62 | +are rejected. Main loops require a fresh run ID. Recovery/evaluation replaces complete arm output; |
| 63 | +v1 resumes only outputs containing the exact unique expected task/run keys. |
| 64 | + |
| 65 | +## Environment |
| 66 | + |
| 67 | +| Variable | Default / meaning | |
| 68 | +|---|---| |
| 69 | +| `ALTIMATE_CMD` | Bun CLI in this repository; honors quoted paths | |
| 70 | +| `DBT_BIN`, `DBT_PYTHON` | `dbt` on PATH; Python beside dbt, or explicitly selected interpreter | |
| 71 | +| `AGENT_MODEL` | Shared Python drivers: `google-vertex-anthropic/claude-haiku-4-5@20251001`; v1 and budget drift shell drivers: `google-vertex/gemini-3.5-flash` | |
| 72 | +| `REFLECTOR_MODEL`, `REVIEWER_MODEL` | Shared default: `google-vertex-anthropic/claude-sonnet-4-6@default`; drift shell reviewer: `google-vertex/gemini-3.1-pro-preview`; independently overridable | |
| 73 | +| `BACKEND` | Historical default `saas`, requiring explicit opt-in; `fake` uses loopback and isolated test users | |
| 74 | +| `ALLOW_REAL_SAAS` | Unset; must equal `1` to permit real workspace access | |
| 75 | +| `WORKSPACE_ID` / `--workspace-id` | Required positive ID for SaaS; binding must resolve to it | |
| 76 | +| `SAAS_CREDS_DIR` | No default; explicit directory containing `home-a/.altimate/altimate.json` and `home-b/...` for SaaS | |
| 77 | +| `RUNS`, `PARALLEL`, `K`, `RUNS_VAL` | Wrapper sample count, concurrency, iterations and validation repetitions | |
| 78 | +| `AGENT_TIMEOUT`, `STAGGER_SECONDS` | Agent deadline (600s), startup spacing (3s) | |
| 79 | +| `BASELINE_FROM` | Corrections driver defaults to `runs/saas-v2/eval/none.jsonl`; set empty to evaluate anew; retain original logs when reusing | |
| 80 | +| `FIX_SRC_ROOT`, `STRONG_MODEL`, `WEAK_MODEL` | Drift matrix checkout (historical `rsi-fix` worktree by default), reflector choices | |
| 81 | +| `BUDGET_REAL_FILE` | Optional real-lesson source; defaults to `runs/corr-main/playbooks/final.md` | |
| 82 | +| `FAKE_DEBUG_TOKEN` | Standalone fake-server admin token; unset disables debug routes; harness generates its own | |
| 83 | + |
| 84 | +SaaS publishing can update an existing **owned** `team-playbook`. Only run it against an intended test |
| 85 | +workspace, with `BACKEND=saas ALLOW_REAL_SAAS=1 WORKSPACE_ID=... SAAS_CREDS_DIR=...` (Python entrypoints use |
| 86 | +`--backend saas --workspace-id ...`). No real SaaS or paid benchmark is needed for self-tests. |
| 87 | + |
| 88 | +Lesson pools/playbooks are deterministic: regenerate with `python3 harness/v1bench/pool.py` then |
| 89 | +`python3 harness/v1bench/make_playbooks.py` from this directory. Committed pools, playbooks, lesson IDs/order, and benchmark treatment defaults match `780168ab18`. |
| 90 | +Historical result Markdown predates these verifier and delivery fixes; it is retained as historical evidence, |
| 91 | +not rescored or claimed as current benchmark results. |
0 commit comments