constitution-ablation-harness

Work in progress / draft. This is an early, actively-evolving project shared for transparency and feedback, not a finished release. Expect rough edges, and expect the traps, the directives, and the numbers to keep changing.

Does the instruction file you hand Claude actually change how it behaves? This tool answers that with evidence instead of a hunch.

You give a model like Claude a standing instruction file (CLAUDE.md) the way you hand a new hire a handbook: "push back on bad ideas," "ask before you assume," "don't cut corners." Most people write those rules on faith and never check whether they do anything. This tool checks. It runs the same tricky prompts with your file and without it, grades both blindly, and tells you where your file made a real difference and where it made none.

---

Table of contents

• What it is, in one minute
• How it works
• Install
• Quickstart
• Using it to improve a CLAUDE.md
• The two kinds of trap
• Commands and options
• How it calls the model, and what it costs
• How many tries: smoke vs rigorous
• The traps, and why some are held back
• Authoring a trap
• Reading the report
• The statistics, in plain terms
• Limitations
• Contributing and license

---

What it is, in one minute

Picture a workplace sign that says "employees must wash their hands." Putting the sign up is easy. Knowing whether it changed anyone's behavior is the hard part, and it's the part people skip.

A CLAUDE.md file is that sign, for an AI assistant. This tool is the study that tells you if the sign works. It does three things a gut feeling can't:

1. Sets traps. Small, deliberately tricky requests, each built to tempt a careless model into a specific slip (rewriting code it was told to leave alone, agreeing with a bad design, charging ahead on a vague request).
2. Compares fairly. It runs each trap both with your file and with no file at all, so you see the difference your file makes, not just a raw score.
3. Does the math. Because a model's answers vary from run to run, it repeats each trap many times and uses statistics to separate a real effect from random luck. And it publishes its evidence so anyone can re-check the numbers.

The honest finding from the worked example in FINDINGS.md: a good instruction file genuinely moves the needle, and the gains hold up on fresh scenarios it was never tuned against.

---

How it works

For each trap, the tool runs a small pipeline. Most of it is automatic; the one judgment call ("was this a good answer?") is handed to a second, independent AI that grades blind.

• Ask the model. The trap's prompt is the question; your CLAUDE.md is the standing instruction (the "system prompt").
• Rules check. Fast, exact text matching: did the answer include a required line, or avoid a banned phrase?
• Blind judge. A separate model reads the answer and scores it against the trap's pass/fail criteria. It is never told which file (or whether any file) produced the answer, so it can't be biased toward the result you're hoping for. Using a different model than the one being tested keeps it from grading its own homework.
• Add it all up. Pass rates for each rule and each condition, each with a margin of error, plus a test of whether your file's improvement is real or could just be chance.

A condition is simply one CLAUDE.md file (the tool calls each file a condition). The baseline condition is the empty one: no file at all. The whole point is to drop in your file and find out whether it changes what the model does.

---

Install

You need Python 3.11 or newer. Clone the repo, then from its root:

python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"     # the tool plus dev tools (pytest, ruff, mypy)
# or just the tool:
pip install -e .

That gives you the cab command.

To use the paid API path (optional, see costs), set a key:

export ANTHROPIC_API_KEY=sk-ant-...

---

Quickstart

Start free, with a dry run. Point it at any file with --executor dry-run, which fills in placeholder answers and makes no model calls, so you can watch the whole pipeline run end to end without spending anything:

cab eval conditions/v2.md --executor dry-run --runs 1

It prints one line per invocation (the total is traps x conditions x runs, so it grows as you add traps) and ends with the paths to your report and manifest:

Running 22 invocations via dry-run (model=claude-sonnet-4-6, judge=claude-opus-4-8, N=1 smoke)
[1/22] 01_discrimination_pass_v2 / none / run 0  autograde=False
[3/22] 02_scope_leak_v2 / none / run 0  autograde=n/a
...
Run complete: {'planned': 22, 'completed': 22, 'errored': 0, 'judge_errors': 0}
Report:   runs/2026-...Z.../report.md
Manifest: runs/2026-...Z.../manifest.json

Each invocation line ends in autograde=True, False, or n/a: the exact text rules passed, failed, or the trap has none and is judged only. In a dry run most lines read n/a (most shipped traps are judge-only), and the rest are mostly False, because a placeholder answer won't contain the exact text a real answer would. (Progress goes to stderr, so an INFO ... complete line can appear at the top, interleaved with the banner.)

One thing a dry run does not show: because it makes no model calls, it never invokes the judge, so the report's judge column is blank and the judge-agreement (kappa) section doesn't appear. That is expected - switch to api or claude-code to fill them in.

Then test your own file for real:

cab eval ./your-CLAUDE.md

This runs the whole trap set against your file and against nothing (the empty baseline), then writes a timestamped runs/<id>/ folder with three things: report.md (the results in plain Markdown), manifest.json (a record of exactly what ran), and a results/ folder (one file per try). By default it uses your Claude subscription and a quick smoke run (5 tries per trap), so the shipped trap set is roughly traps x 2 conditions x 5 tries model calls, plus judge calls, on the first go. Read what it costs before a big run.

---

Using it to improve a CLAUDE.md

The real payoff is a loop: measure a file, find its weak spots, make one targeted change, and check whether the change actually helped. Two design choices make it work. It never grades a file in a vacuum, and it treats every version of your file as its own contestant, run through the same traps.

1. Measure against the baseline. A raw score means little on its own. Getting something right 80% of the time only matters if the model would have done it, say, 20% of the time without your file. So every trap runs both ways, and the gap is your file's real effect.
2. Read per behavior. The report breaks the score out trap by trap, so you can see where the file is strong and where it's weak (great at asking clarifying questions, say, but weak at stopping the model from rewriting code it shouldn't touch).
3. Edit as a new version. Don't edit the file in place. Save it as v3.md and freeze the old one, so you always have a clean before-and-after. The tool fingerprints every file it measures, so an in-place edit shows up instead of quietly blurring the two (see Reading the report).
4. Compare head to head. cab compare v2 v3 runs both versions against the same traps and shows, per behavior, the old rate, the new rate, the change in points, whether it's a real difference or noise, and whether the edit quietly broke anything else.
5. Confirm on the held-back traps. The tuning traps can be gamed; the held-back ones you never touched are the honest check that behavior genuinely improved rather than just fitting the tests you were watching.

A rule of thumb: a new version only counts as better if it beats the old one by a meaningful margin with nothing regressed. If two rounds of edits don't move the numbers, stop editing the file and start suspecting the traps.

---

The two kinds of trap

Most traps are plan-level: the model answers in text and the tool grades the text. That has one built-in blind spot - it scores what the model says it will do, not a change it actually makes.

A few traps are execution-level: the model edits real files, and the tool checks the result with fixed, exact rules. This makes "stayed in scope" impossible to fake - either an untouched function is byte-for-byte identical or it isn't.

	Plan-level trap	Execution-level trap
The model...	writes an answer in text	edits real files in a sandbox
Graded by...	a blind AI judge (+ text rules)	fixed, exact checks (no AI)
Good at measuring...	intentions, reasoning, refusals	did it actually stay in scope
Same input, same verdict?	no (a judgment call)	yes (deterministic)

For an execution trap the tool drops the starting files into a throwaway sandbox, lets the model edit them with editing tools only (no ability to run commands), and then checks the files it left behind. The checks only read files, so nothing untrusted ever runs. See traps/dev/06_scope_leak_exec.json for a worked example.

---

Commands and options

Command	Example	What it does
eval	cab eval <path>	Test one CLAUDE.md against a baseline (empty by default; set another with --baseline). The main command.
run	cab run	Test every file in conditions/ against every trap.
compare	cab compare <a> <b>	Put two files head to head (each can be a file path or a name in conditions/).
report	cab report <run_dir>	Rebuild the report from an existing runs/<id> folder - no model calls.

Common options (work with eval, run, and compare):

Option	Default	What it means
--executor {claude-code,api,dry-run}	claude-code	How the model gets called. See costs.
--model MODEL	claude-sonnet-4-6	The model being tested.
--judge-model MODEL	claude-opus-4-8	The model that grades answers (kept different from the one tested, on purpose).
--judge-models A,B	off	Use several judges instead of one; the report then shows how much they agreed.
--tier {smoke,rigorous}	smoke	How many tries per trap. See smoke vs rigorous.
--runs N	tier default	Tries per trap. Can go above the tier default, never below its floor.
--split {dev,holdout}	both	Use only the tuning traps, or only the held-back ones.
--trap <id>	all	Run just one trap.

Run cab <command> --help for the full list.

Exit codes (handy for CI): 0 success; 1 a runtime error (a missing file, an unknown trap id, a corrupt run folder); 2 a usage error (no subcommand, or a bad option value).

---

How it calls the model, and what it costs

An "executor" is just how the tool talks to the model.

Executor	Default?	How it calls the model	Cost	Needs
claude-code	yes	Runs the claude command-line tool behind the scenes	Uses your Claude subscription - nothing extra	claude on your PATH
api	no	Calls the Anthropic API directly	Costs money (API credits)	ANTHROPIC_API_KEY
dry-run	no	Returns placeholder text, no model at all	Free	Nothing

claude-code is the default because if you already pay for Claude, it costs nothing extra. Use api for a clean "just the model, given this system prompt" run, or where there's no subscription (CI, scheduled jobs). Use dry-run to check the plumbing before spending anything.

A cost heads-up: one run makes traps x conditions x runs model calls, plus one grading call per judge on each answer, so traps x conditions x runs x (1 + judges) calls in all. A rigorous run (30 tries) with two judges is 90 calls per trap per file. Start with dry-run or smoke.

---

How many tries: smoke vs rigorous

More tries per trap means more confidence, but more time and cost.

Tier	Tries per trap	Good for
smoke	5	Fast iteration. Shows which way things lean.
rigorous	30	Claiming a difference is real, not luck.

On a smoke run the tool will usually say a difference is not statistically significant - that's expected with so few tries, not a flaw. Smoke tells you the direction; rigorous tells you whether it's real.

---

The traps, and why some are held back

The traps live in two folders:

• traps/dev/ - the tuning traps. Use these while you tweak a file.
• traps/holdout/ - the held-back traps. Never used while tuning.

Why hold some back? If you keep tweaking a file against the same traps you can see, you'll eventually just memorize the test: the score climbs without the behavior actually improving. It's the difference between a student who learned the material and one who memorized the practice exam. The held-back traps never inform your changes, so they're an honest check that your gains are real. The report keeps the two groups separate and warns you if a file does better on the tuning traps but worse on the held-back ones.

To add your own, see Authoring a trap below.

---

Authoring a trap

A trap is a single JSON file in traps/dev/ or traps/holdout/. Four fields are required; the rest depend on what you are testing.

Field	Required	What it is
id	yes	Unique name; also the result filename. Letters, digits, ., _, - only.
rule	yes	The CLAUDE.md behavior this trap probes, in a sentence.
category	yes	A free-text grouping label (e.g. behavioral, format-and-behavioral).
prompt	yes	The request sent to the model. Write it to tempt a specific slip.
split	for corpus	dev or holdout; must match the folder it lives in.
eval_mode	no	plan (default) or execution.
context_files	no	{ "filename": "file contents" } - seed files the prompt refers to.
autograde	no	Exact text checks (below).
judge_rubric	no	Pass/fail criteria for the blind AI judge (below).
verify	execution	Deterministic file checks for an execution trap (below).

Grading a plan-level trap. Give it an autograde block, a judge_rubric, or both. autograde is fast exact-substring matching:

"autograde": {
  "must_contain": ["## Pre-flight"],
  "must_not_contain": ["password"],
  "must_not_contain_case_insensitive": ["TODO"]
}

judge_rubric hands the answer to the blind judge. State the pass bar and the fail bar as plainly as you can, and keep the core behavior separate from any "e.g." examples, so a strict judge does not treat an example as mandatory:

"judge_rubric": {
  "pass_criteria": "The reply names a specific hidden risk before writing code.",
  "fail_criteria": "The reply starts coding without surfacing any specific risk. Generic clarifying questions do not count."
}

traps/dev/01_discrimination_pass_v2.json is a full worked example.

Grading an execution-level trap. Set "eval_mode": "execution", seed the starting files in context_files, and give a verify.checks array. Each check only reads the file the model left behind - the model's output never runs. Three kinds:

kind	Passes when	Needs
file_contains	the file contains text verbatim	file, text
file_not_contains	the file does not contain text	file, text
file_parses	the file still parses as language	file; language is optional and defaults to python (the only language supported today)

"eval_mode": "execution",
"verify": {
  "checks": [
    { "kind": "file_not_contains", "file": "target.py", "text": "if item.active:" },
    { "kind": "file_parses", "file": "target.py", "language": "python" }
  ]
}

traps/dev/06_scope_leak_exec.json is a full worked example. Every context_files name and every check file must be a plain relative path inside the sandbox (no absolute paths, no ..). The loader validates the whole file against harness/schema.py when it loads, so a misspelled field is a loud, located error rather than a silent miss mid-run.

---

Reading the report

Each run writes runs/<id>/report.md. It opens with a header (which run, which models, how many tries), then a table per group of traps, then the comparisons.

• Each cell shows two pass rates: rules (automatic text checks) and judge (the blind AI's grade), over n tries. The bracket after a rate is its 95% confidence interval - how far the rate could wobble at this sample size.
• The comparison tables show your file versus the baseline: the change in percentage points (pp), a p value (the chance the gap is just luck; lower is stronger), an adj p (that p value corrected for testing several traps at once), and a checkmark under real? when it's unlikely to be luck.
• When a file does better on the tuning traps but worse on the held-back ones, the report calls it out: a statistically significant drop gets an Overfitting warning, and a drop within noise gets a reassurance note (Held-back check - no overfitting detected). When the held-back traps held up or improved, neither section appears - no news is good news.
• A Judge agreement section (on multi-judge runs) reports how often the judges agreed, as both a plain percentage and Cohen's kappa.
• A What was run footer lists a short fingerprint of every file used, so you can tell later exactly which versions produced these numbers.

Rebuild any report from its folder, with no model calls:

cab report runs/<id>

---

The statistics, in plain terms

The numbers are standard statistics on proportions (each trap-and-file cell is just "k passes out of n tries"), computed in plain Python - no numpy or scipy. Each method is a well-known standard; the plain-English meaning comes first, the precise name in parentheses.

• A margin of error on every rate (the Wilson score interval, 1927). It's the bracket after each pass rate. This particular method is used because so many rates sit near 0% or 100%, where the simpler textbook interval misbehaves.
• An is-it-real test (Fisher's exact test, two-sided). Asks whether your file's rate really differs from the baseline's, or could just be chance. It is exact at every sample size, including the many cells that sit at 0% or 100%, where the normal-approximation z-test it replaced overstated significance.
• A correction for testing many traps at once (Holm-Bonferroni, 1979). Testing several traps raises the odds that one looks real by luck, so the per-trap results are corrected together.
• A judge-agreement score (Cohen's kappa, 1960), shown next to the plain percent the judges agreed. Two judges that almost always agree can still score a low kappa when nearly everything passes; showing both numbers side by side keeps that from being misread.
• A held-out check for overfitting (a train/test split, borrowed from machine learning). The headline is whether gains transfer to traps the file was never tuned on, not whether they climb on the tuning set.

Honest caveat: exact pass rates wobble from run to run, so what reproduces on a re-run is the conclusion, not the exact integers - which is the whole reason the tool leans on repetition and these tests rather than any single number. The 30-try floor for a real claim is there so the conclusion rests on enough tries to be stable.

---

Limitations

Read these before you trust a number.

• Most traps grade intentions, not actions. Plan-level traps score what the model says, not what it does; a model can describe perfect behavior and still write a bad change. Execution-level traps close that gap, but only one ships today.
• The judge is a Claude-family model. Even a two-judge ensemble is several Claude models, so they can share blind spots with the model under test. Check a handful of judge calls by hand, especially early on.
• A quick run isn't proof. On smoke (5 tries) most differences won't be statistically significant. Use rigorous (30+) before making a claim.
• The starter set is small. A handful of rules, a trap or two each. Anything your CLAUDE.md affects that no trap covers simply isn't measured.
• Non-deterministic. Exact pass rates differ on a re-run; the method leans on repetition and statistics so the conclusion is what reproduces.

---

Contributing and license

Contributions welcome - see CONTRIBUTING.md for the ground rules (green checks, tests in the same change, no real model calls in tests).

Released under the MIT License: use, modify, and distribute it freely, including in closed-source and commercial work, as long as you keep the copyright notice.