[mjunaidca]

A live walk of the published 0.0.59 reported that add-sources did not fire on its headline prompt. The tempting repair is to reword the description until it does — a guess. This is the instrument that replaces the guess, plus what it found when pointed at the actual question.

What it does

packages/ksor/src/evals/skill-triggers.agent.test.ts: N runs per phrase, in a fresh scaffold with all three skills present, graded on which skill the agent actually invoked — read from the whole transcript, not the first tool call, so an agent that reads a file before deciding is not scored a miss. Five phrases, including two controls: one where a different skill must win, one where none should.

Gated like the content eval (logged-in claude locally, CLAUDE_CODE_OAUTH_TOKEN in CI) and announces its own skip. Reported, never gating — a model is stochastic and a threshold over a handful of runs flakes.

What it found

The model is a column, because the answer depends on it. Same phrase, same scaffold, same harness:

phrase	sonnet-5	opus-5
own-words — "add this to the knowledge base: …"	3/3	3/3
from-memory — nobody wrote it down	3/3	3/3
get-started → intake-interview (control)	3/3	3/3
stay-silent → nothing fires (control)	3/3	3/3
file-in-repo — "here is src/policy.txt, add it under finance/"	3/3	0/3

That is why the walk (default model, one run) and my first probe (Sonnet, pinned) disagreed. Neither was wrong; neither alone measured the trigger.

The obvious repair does not work. I added a clause naming exactly that shape — "points at a file already in the repo and says where it belongs", shipped as 2.1.0 — and re-measured on the model that missed: 0/3 before, 0/3 after. So the cause is not the wording. An instruction concrete enough to act on gets acted on, and no skill is consulted. The clause is reverted (add-sources stays 2.0.0): it is resident context in every session and bought nothing a measurement can see. The negative result is recorded beside the rows it explains, per the invariant about recording negative results next to the constant they explain.

The question this leaves — owner's, not mine

On the one path where the skill does not fire, the record still came out well-formed, because the emitted AGENTS.md carries the same rules and is always resident. The content eval said the same from the other side: both arms passed every behavioural gate. So is add-sources earning its resident cost on that path, or is AGENTS.md already doing the work? AGENTS.md's own rule is that a skill nobody can show winning is deleted. The note in the harness header says how to answer it: a fixture the two arms score differently, or delete the skill — not another reworded description.

Also fixed

The CLI can emit raw control characters inside its JSON, which JSON.parse rejects outright — the harness lost the entire transcript to a stray byte. It now falls back to a scrubbed parse. Hit by hand while probing; it would have shown up as a flaky "did not return JSON".

Verified

unit 1702, integration 749 passed / 34 skipped, agent tier unarmed 5 passed / 4 skipped (announces its skip), typecheck 0, guard ok, corpus ok, format clean. About $17 of real agent runs, all recorded in TRIGGER_BASELINE.