Skip to content

Commit 71eed27

Browse files
Merge origin/main into fix/test-leaked-skill-syncs
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0174HUYiaceApqHP7EKMNgnb
2 parents 59e2052 + 5bfb231 commit 71eed27

368 files changed

Lines changed: 53235 additions & 409 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitguardian.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,11 @@ paths-ignore:
2020
# (PWD here is the unix env var for present working directory, not a
2121
# password). Test exercises symlink/cwd resolution.
2222
- packages/opencode/test/cli/tui/thread.test.ts
23+
# `learn` redaction and guard-rail tests: synthetic credentials (`hunter2`,
24+
# AWS's documented `AKIAIOSFODNN7EXAMPLE`, fake JWT/basic-auth strings) that
25+
# verify secrets are scrubbed before signals are stored or sent to a model.
26+
- packages/opencode/test/altimate/learn/guardrail-corpus.test.ts
27+
- packages/opencode/test/altimate/learn/review-redaction.test.ts
28+
- packages/opencode/test/altimate/learn/digest.test.ts
29+
- packages/opencode/test/altimate/learn/import-reviews.test.ts
30+
- packages/opencode/test/altimate/learn/sensitive-data.test.ts

‎.github/workflows/ci.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,11 @@ jobs:
5252
- 'bun.lock'
5353
- 'package.json'
5454
- 'tsconfig.json'
55+
# altimate_change start — the Typecheck job runs `bun turbo typecheck`: turbo.json
56+
# defines that task and bunfig.toml shapes the install it runs on.
57+
- 'turbo.json'
58+
- 'bunfig.toml'
59+
# altimate_change end
5560
drivers:
5661
- 'packages/drivers/src/**'
5762
- 'packages/opencode/src/altimate/native/connections/**'
@@ -124,6 +129,47 @@ jobs:
124129
run: bun script/check-tracker-leaks.ts
125130
# altimate_change end
126131

132+
# altimate_change start — typecheck every workspace on PRs and on main.
133+
# ---------------------------------------------------------------------------
134+
# `bun turbo typecheck` ran only in release.yml and in the local pre-push hook,
135+
# and the TypeScript job below runs tests, not the compiler. So nothing in CI
136+
# caught two PRs that each passed on their own leaving main with a type error
137+
# together; it surfaced as every developer's pre-push failing after they
138+
# merged main, and would have failed the next release.
139+
typecheck:
140+
name: Typecheck
141+
needs: changes
142+
if: needs.changes.outputs.typescript == 'true' || github.event_name == 'push'
143+
runs-on: ubuntu-latest
144+
timeout-minutes: 15
145+
permissions:
146+
contents: read
147+
steps:
148+
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
149+
with:
150+
persist-credentials: false
151+
152+
- uses: oven-sh/setup-bun@ecf28ddc73e819eb6fa29df6b34ef8921c743461 # v2
153+
with:
154+
bun-version: "1.3.14"
155+
156+
- name: Cache Bun dependencies
157+
uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
158+
with:
159+
path: ~/.bun/install/cache
160+
key: bun-${{ runner.os }}-${{ hashFiles('bun.lock') }}
161+
restore-keys: |
162+
bun-${{ runner.os }}-
163+
164+
- name: Install dependencies
165+
run: bun install
166+
167+
# The same command as release.yml's gate and the pre-push hook, so all
168+
# three agree on what "typechecks" means.
169+
- name: Typecheck
170+
run: bun turbo typecheck
171+
# altimate_change end
172+
127173
# Main TypeScript tests — excludes driver E2E tests (separate job) and
128174
# cloud credential tests (local-only).
129175
# ---------------------------------------------------------------------------

‎docs/docs/configure/learn.md‎

Lines changed: 586 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/docs/configure/warehouses.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,9 @@ Each warehouse has a key (the connection name) and a config object:
2323
!!! tip
2424
Use `{env:...}` substitution for passwords and tokens so you never commit secrets to version control.
2525

26+
!!! note "Where saved passwords go"
27+
When the agent adds a connection for you, passwords, keys and tokens are kept in your OS credential store (macOS Keychain, Windows Credential Manager, or libsecret on Linux), never in `connections.json`. If no credential store is available, the credential is not saved: the connection works for that session only, and Altimate Code says so. In that case set an `ALTIMATE_CODE_CONN_<NAME>` environment variable to the connection's full config as JSON (not just the password); for a name containing `-`, set it with `env`, since a shell assignment cannot use that character. Removing a saved connection removes its credentials from the store, unless this project's connections file or an environment variable still defines a connection by that name. Saving a connection again under the same name drops the credentials it no longer uses once the new config is written. Credentials are stored by connection name, so a project file that defines a connection of the same name without its own password was using the saved connection's, and loses it when that one is removed. If the store refuses a removal, Altimate Code says which credentials remain.
28+
2629
## Snowflake
2730

2831
```json
@@ -50,6 +53,7 @@ Each warehouse has a key (the connection name) and a config object:
5053
| `database` | No | Database name |
5154
| `schema` | No | Schema name |
5255
| `role` | No | User role |
56+
| `client_session_keep_alive` | No | Keep the session alive while Altimate Code is running (default `true`). A connection Snowflake closes anyway is reopened automatically. `USE`, `ALTER SESSION` and `SET` settings are applied again to the reopened session. A single read-only query interrupted by the close is run again; a statement that changes data, or calls a sequence, is not, because it may already have run, and the error says so. Temporary tables and an open transaction cannot be restored: statements issued before the reopen report that instead of running. |
5357

5458
### Key-pair authentication
5559

@@ -68,6 +72,26 @@ Each warehouse has a key (the connection name) and a config object:
6872
}
6973
```
7074

75+
### Browser single sign-on
76+
77+
Set `"authenticator": "externalbrowser"` (and no password) to sign in through your identity provider.
78+
79+
```json
80+
{
81+
"snowflake-sso": {
82+
"type": "snowflake",
83+
"account": "xy12345.us-east-1",
84+
"user": "jane.doe@example.com",
85+
"authenticator": "externalbrowser",
86+
"warehouse": "COMPUTE_WH"
87+
}
88+
}
89+
```
90+
91+
When a connection opens, Snowflake's sign-in page opens in your default browser, and Altimate Code shows **Waiting for sign-in** with a link to the same page in case no browser tab appeared. Snowflake waits up to two minutes for the sign-in. If it isn't completed in that time, the connection fails with a message saying so; complete the sign-in, then try again. If you'd rather not sign in through the browser, use [key-pair authentication](#key-pair-authentication).
92+
93+
A connection that does not use the browser fails after two minutes without an answer from Snowflake, with a message pointing at network, VPN or proxy settings, instead of retrying silently.
94+
7195
## BigQuery
7296

7397
```json

‎docs/docs/reference/troubleshooting.md‎

Lines changed: 25 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -245,13 +245,35 @@ Or manually compact in the TUI: leader + `Shift+C`.
245245

246246
## Debug Mode
247247

248-
Run with full debug output:
248+
When something goes wrong that is hard to reproduce (a hang, Altimate Code closing unexpectedly, a connection that fails sometimes), turn on debug mode, reproduce the problem, then create a diagnostic report.
249+
250+
**1. Turn on debug mode** for the session where you will reproduce the problem:
249251

250252
```bash
251-
altimate --print-logs --log-level DEBUG 2>debug.log
253+
# macOS / Linux
254+
ALTIMATE_DEBUG=1 altimate
255+
256+
# Windows (PowerShell)
257+
$env:ALTIMATE_DEBUG = "1"; altimate
258+
```
259+
260+
In debug mode the log records every tool call as it starts and ends, and while one is running, a line every 15 seconds naming what is still in progress. Lines are written as they happen, so if Altimate Code hangs or closes, the log still shows what it was doing.
261+
262+
**2. Create the report** after the problem has happened:
263+
264+
```bash
265+
altimate debug bundle
252266
```
253267

254-
Then share `debug.log` when reporting issues.
268+
This writes `altimate-debug-report-<time>.md` in the current folder (use `--output <path>` to choose another place). It lists the problems it detected first, then the evidence: version and install method, operating system and terminal, warehouse connections (names, types, sign-in method and which fields are set, never values), MCP servers, a summary of the log, and the last 300 log lines. It also checks that the Snowflake and Databricks warehouses, the Altimate API, the telemetry endpoint (when telemetry is on) and the model catalogue this installation uses are reachable over HTTPS. Add `--no-network` (or `--network=false`) and the command makes no network requests at all, telemetry included. It does not start your project (plugins, language servers), so it still works when that is what hangs. The file is created readable only by you, where the filesystem supports it.
269+
270+
Passwords, keys, tokens and other secret settings (in `name=value`, JSON and connection-URL forms), email addresses, URL parameters, your home folder, your user name and your computer's name are removed. This is pattern-based and can miss a secret in an unusual format, so read the report before you send it. It stays on your machine until you send it to Altimate support yourself.
271+
272+
To save the raw logs to a file instead:
273+
274+
```bash
275+
altimate --print-logs --log-level DEBUG 2>debug.log
276+
```
255277

256278
## Getting Help
257279

‎docs/docs/usage/cli.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,11 @@ altimate --agent analyst
4040
| `import` | Import session data |
4141
| `session` | Session management |
4242
| `link` | Link this project to an Altimate workspace (pilot, requires `ALTIMATE_WORKSPACE=1`) |
43+
| `workspace` | Show, refresh, sync or unlink this project's workspace from a shell (pilot, requires `ALTIMATE_WORKSPACE=1`) |
4344
| `trace` | List and view session traces (recordings of agent sessions) |
4445
| `github` | GitHub integration |
4546
| `pr` | Pull request tools |
47+
| `debug` | Troubleshooting tools -- `debug bundle` writes a diagnostic report to send to support; see [Debug Mode](../reference/troubleshooting.md#debug-mode) |
4648
| `upgrade` | Upgrade to latest version |
4749
| `uninstall` | Uninstall altimate |
4850

@@ -51,6 +53,11 @@ altimate --agent analyst
5153
Workspace features are off unless `ALTIMATE_WORKSPACE=1` is set. With it:
5254

5355
- `altimate-code link` links the current project to a workspace (or creates one). The sidebar then names the workspace and shows how many memories are not yet synced and when skills last synced.
56+
- From a shell, script or CI job, without the TUI:
57+
- `altimate-code link --workspace <name|id>` links to an existing workspace (an id is matched before a name), and `altimate-code link --create [name]` creates one (named after the repo by default) and links to it. Neither prompts. If the project is already linked to a different workspace they refuse unless `--yes` is passed. Running `--create` again when the project is already linked to a workspace of that name creates nothing, so it is safe in a devcontainer's setup command; if another workspace already has that name it refuses and names its id, unless `--allow-duplicate` is passed. Linking to the workspace the project is already linked to confirms that link on this machine, which a fresh clone needs before `workspace sync` sends its memory.
58+
- `altimate-code workspace status [--json]` shows the linked workspace and, when known, how many saved memories have not reached it and when skills last synced.
59+
- `altimate-code workspace refresh` pulls the workspace's skills (memory is loaded by the next session started in the project); `altimate-code workspace sync` sends memory the workspace has not received; `altimate-code workspace unlink [--yes]` detaches the project (`--yes` is required without a terminal).
60+
- These commands accept `--directory`. Exit codes: `0` done, `1` failed (including when the workspace service could not be reached), `2` a request to change (not signed in, a re-link or unlink without `--yes` and no terminal to confirm on, or a `sync` to a link not yet confirmed on this machine), `3` the project is not linked. With `--json`, `ok` is `true` exactly when the exit code is `0`.
5461
- `/workspace` in the TUI opens a menu: **Refresh** pulls the workspace's skills and memory into this project, **Sync** re-sends local memory the workspace never received, **Open in browser** (when a web URL is available) shows the workspace on the web, **Switch workspace** relinks the project, **Unlink** detaches it. In a project that is not linked yet, it offers **Link to a workspace** instead.
5562
- `altimate-code skill publish <name>` uploads a project skill to the linked workspace; see [Skills](../configure/skills.md#cli-commands).
5663
- In an ordinary session the agent is told every turn which workspace the project is linked to — or that none is, or that the link could not be verified just now. In an extension-pinned session it is told the pinned workspace instead, and that it differs from the project's own link. Either way "which workspace am I in?" has an answer, and the agent is told not to confuse it with a Databricks workspace or an IDE workspace folder.
@@ -79,6 +86,7 @@ Configuration can be controlled via environment variables:
7986
| Variable | Description |
8087
| ----------------------------- | ---------------------------- |
8188
| `ALTIMATE_CLI_CONFIG` | Path to custom config file |
89+
| `ALTIMATE_DEBUG` | `1` turns on debug mode: every tool call and a heartbeat for long ones are recorded in the log (see [Debug Mode](../reference/troubleshooting.md#debug-mode)) |
8290
| `ALTIMATE_CLI_CONFIG_DIR` | Custom config directory |
8391
| `ALTIMATE_CLI_CONFIG_CONTENT` | Inline config as JSON string |
8492
| `ALTIMATE_CLI_GIT_BASH_PATH` | Path to Git Bash (Windows) |

‎docs/mkdocs.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,7 @@ nav:
9696
- Memory Tools: data-engineering/tools/memory-tools.md
9797
- Custom Tools: configure/tools/custom.md
9898
- Skills: configure/skills.md
99+
- Learn: configure/learn.md
99100
- Commands: configure/commands.md
100101
- Validators: data-engineering/validators.md
101102
- Trace: configure/trace.md
@@ -177,6 +178,7 @@ nav:
177178
- Memory Tools: data-engineering/tools/memory-tools.md
178179
- Custom Tools: configure/tools/custom.md
179180
- Skills: configure/skills.md
181+
- Learn: configure/learn.md
180182
- Commands: configure/commands.md
181183
- Validators: data-engineering/validators.md
182184
- Trace: configure/trace.md
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
# RSI research harness
2+
3+
Research tooling only. Model-driven suites incur provider charges; the self-tests below do not.
4+
The learning loops, v1 delivery, bootstrap, and drift suites require the **`learn` features from
5+
PR #1405**. This harness-only branch does not supply them. Use a checkout containing those
6+
features through `ALTIMATE_CMD`; changing product code is outside this experiment.
7+
The verifier, fake backend, and local `none`/`gold`/`playbook` evaluation arms do not need `learn`.
8+
9+
Prerequisites: Bun, Git, repository dependencies installed for the selected CLI, and Python
10+
3.10+ for the dbt environment (harness syntax supports 3.9+). The local checks used Python 3.11.
11+
For example, from the repository root:
12+
13+
```bash
14+
python3.11 -m venv /tmp/rsi-dbt
15+
/tmp/rsi-dbt/bin/pip install dbt-core==1.12.5 dbt-duckdb==1.11.0
16+
export DBT_BIN=/tmp/rsi-dbt/bin/dbt
17+
export DBT_PYTHON=/tmp/rsi-dbt/bin/python
18+
```
19+
20+
Pin package versions and record the CLI commit and model IDs alongside any reported results.
21+
The shared default CLI is this repository's `packages/opencode/src/index.ts`; baseline and fix
22+
comparison wrappers retain their historical `rsi-base` and `rsi` worktrees. An override is a
23+
shell-quoted command string, e.g. `ALTIMATE_CMD="bun run --conditions=browser '/path with spaces/src/index.ts'"`.
24+
Model-driven runs need provider authentication usable from the isolated homes. Account/org policies
25+
may block Claude, including Claude on Vertex; choose models your account can use. The harness checks
26+
requested model IDs before evaluation and treats timeouts, errors and incomplete turns as failed runs.
27+
28+
## Running suites
29+
30+
Free local checks (from the repository root):
31+
32+
```bash
33+
experiments/rsi-workspace/demo/run_task.sh selftest
34+
python3 experiments/rsi-workspace/fake-backend/selftest.py
35+
python3 experiments/rsi-workspace/harness/selftest.py
36+
python3 experiments/rsi-workspace/harness/v1bench/selftest.py
37+
```
38+
39+
Model-driven suites, from `experiments/rsi-workspace/harness/`:
40+
41+
```bash
42+
bash run_all.sh my-ci-loop # train / validation gate / publish / heldout arms
43+
bash run_corrections.sh my-corrections # simulated teammate corrections / publish / eval
44+
python3 eval.py --arm gold --runs 1 --run-id my-eval --out runs/my-eval/eval/gold.jsonl
45+
bash run_arms.sh my-ci-loop # rerun final arms of an existing completed loop
46+
python3 publish_replace.py runs/my-ci-loop # recover publishing using supported publish/update
47+
bash budget/run_budget.sh # rule-budget arms
48+
bash budget/run_budget2.sh # scope/conflict arms
49+
bash budget/run_drift.sh my-drift "${REFLECTOR_MODEL:-google-vertex/gemini-3.5-flash}"
50+
FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix.sh
51+
FIX_SRC_ROOT=/path/to/fix-checkout bash budget/run_drift_matrix2.sh
52+
bash v1bench/run_baselines.sh # n50-short / all-50 / all-300 / all-1000
53+
bash v1bench/run_all_v1.sh # retrieval, vague task, topic, bootstrap, drift suites
54+
bash v1bench/run_fix_v1.sh # focused fix comparison (ALTIMATE_CMD selects checkout)
55+
```
56+
57+
See [v1 suite plan](harness/v1bench/PLAN.md), [topic switch](harness/v1bench/topic_switch/README.md),
58+
[vague tasks](harness/v1bench/vague_tasks/README.md), [demo](demo/README.md), and
59+
[fake backend](fake-backend/README.md) for task definitions and additional switches.
60+
`fake-backend/demo.sh` runs the model-driven two-user workspace demo.
61+
IDs must be simple names (letters/digits plus `.`, `_`, `-`); empty, absolute and traversing IDs
62+
are rejected. Main loops require a fresh run ID. Recovery/evaluation replaces complete arm output;
63+
v1 resumes only outputs containing the exact unique expected task/run keys.
64+
65+
## Environment
66+
67+
| Variable | Default / meaning |
68+
|---|---|
69+
| `ALTIMATE_CMD` | Bun CLI in this repository; honors quoted paths |
70+
| `DBT_BIN`, `DBT_PYTHON` | `dbt` on PATH; Python beside dbt, or explicitly selected interpreter |
71+
| `AGENT_MODEL` | Shared Python drivers: `google-vertex-anthropic/claude-haiku-4-5@20251001`; v1 and budget drift shell drivers: `google-vertex/gemini-3.5-flash` |
72+
| `REFLECTOR_MODEL`, `REVIEWER_MODEL` | Shared default: `google-vertex-anthropic/claude-sonnet-4-6@default`; drift shell reviewer: `google-vertex/gemini-3.1-pro-preview`; independently overridable |
73+
| `BACKEND` | Historical default `saas`, requiring explicit opt-in; `fake` uses loopback and isolated test users |
74+
| `ALLOW_REAL_SAAS` | Unset; must equal `1` to permit real workspace access |
75+
| `WORKSPACE_ID` / `--workspace-id` | Required positive ID for SaaS; binding must resolve to it |
76+
| `SAAS_CREDS_DIR` | No default; explicit directory containing `home-a/.altimate/altimate.json` and `home-b/...` for SaaS |
77+
| `RUNS`, `PARALLEL`, `K`, `RUNS_VAL` | Wrapper sample count, concurrency, iterations and validation repetitions |
78+
| `AGENT_TIMEOUT`, `STAGGER_SECONDS` | Agent deadline (600s), startup spacing (3s) |
79+
| `BASELINE_FROM` | Corrections driver defaults to `runs/saas-v2/eval/none.jsonl`; set empty to evaluate anew; retain original logs when reusing |
80+
| `FIX_SRC_ROOT`, `STRONG_MODEL`, `WEAK_MODEL` | Drift matrix checkout (historical `rsi-fix` worktree by default), reflector choices |
81+
| `BUDGET_REAL_FILE` | Optional real-lesson source; defaults to `runs/corr-main/playbooks/final.md` |
82+
| `FAKE_DEBUG_TOKEN` | Standalone fake-server admin token; unset disables debug routes; harness generates its own |
83+
84+
SaaS publishing can update an existing **owned** `team-playbook`. Only run it against an intended test
85+
workspace, with `BACKEND=saas ALLOW_REAL_SAAS=1 WORKSPACE_ID=... SAAS_CREDS_DIR=...` (Python entrypoints use
86+
`--backend saas --workspace-id ...`). No real SaaS or paid benchmark is needed for self-tests.
87+
88+
Lesson pools/playbooks are deterministic: regenerate with `python3 harness/v1bench/pool.py` then
89+
`python3 harness/v1bench/make_playbooks.py` from this directory. Committed pools, playbooks, lesson IDs/order, and benchmark treatment defaults match `780168ab18`.
90+
Historical result Markdown predates these verifier and delivery fixes; it is retained as historical evidence,
91+
not rescored or claimed as current benchmark results.

0 commit comments

Comments
 (0)