Results from testing Coding-Agent Skill Distillation (CASD), the method from "Coding Agents are Strong Prompt Optimizers" (arXiv 2609.26261). We tried it first on tasks the paper did not cover, then on one of the paper's own benchmarks.
- Run an agent on a set of practice tasks and keep every log.
- Give the logs to a coding agent with one fixed instruction. It writes a "skill" file: a page of rules for that kind of task.
- Use the skill file as the agent's prompt on new tasks.
No training is involved. The paper reports an average gain of 16.6 points in pass rate across four benchmarks, with GPT-5.4-mini as the model under test.
| # | Model under test | Tasks | No skill | With skill | Change in pass rate | Output tokens with skill | Report |
|---|---|---|---|---|---|---|---|
| 1a | Muse Glimmer 30B | BigCodeBench-Hard, 98 tasks | 27.2% | 28.2% | +1.0, noise | 18% more | 01 |
| 1b | Muse Glimmer 30B | Same tasks, the paper's protocol | 28.6% | 29.6% | +1.0, noise | 21% more | 01 |
| 2 | Claude Opus 4.8 | 10 real bugs in element-web (SWE-bench Pro) | 100% | 90% | -10, not shown | 26% fewer | 02 |
| 3 | Claude Haiku 4.5 | The same 10 bugs | 66.7% | 56.7% | -10, not shown | 21% fewer | 03 |
| 4 | Claude Haiku 4.5 | SpreadsheetBench-Verified, 50 tasks | 66.0% | 74.0% | +8.0 | 11% more | 04 |
| Paper: GPT-5.4-mini | SpreadsheetBench-Verified, 50 tasks | 39.3% | 51.3% | +12.0 |
"Noise" and "not shown" mean the difference is within what chance alone produces at that test size.
- The method worked on the paper's kind of task. On SpreadsheetBench the skill raised Haiku 4.5 from 66% to 74%. That is two-thirds of the paper's gain. Every skill pass beat every no-skill pass. It is likely real but not nailed down at 50 tasks.
- It did not raise the pass rate on real bug fixing, with a strong model (Opus 4.8) or a small one (Haiku 4.5).
- On bug fixing it made runs cheaper instead. Output tokens fell 21 to 26% and time fell 42 to 44%. The saving came from the model checking its work less, and on a few bugs that missing check mattered.
- A model that passes every practice task gives the skill writer nothing to learn from. Opus 4.8 passed all 22 practice bugs, so its skills could only say "do less". With them, 3 of 30 runs failed where none had failed before.
- Smaller models follow a skill loosely. Muse Glimmer printed the line the skill asked for in 87% of runs while the habit behind it barely changed. Haiku 4.5 wrote the checklist its skill asked for in 15 of 30 runs.
- The benchmark decided more than the method did. On BigCodeBench-Hard, 54 of 98 tasks could not pass under any prompt, because the hidden tests check things the task text never states.
Our reading: a skill helps when tasks share habits that carry from one task to the next, such as "read the raw cells" or "always save a file". Each bug is its own problem, and a general rule did not stop a model from missing one detail of the next bug. The paper tested only tasks of the first kind.
These are our mistakes, not the method's. Each cost time and each is described in the report where it happened.
| Mistake | What it cost | Report |
|---|---|---|
| Picked a benchmark without reading its failures first | 882 runs that could not show a gain | 01 |
| Picked a model too strong for the tasks | A test with no room for the pass rate to rise | 02 |
| Ran on a laptop where the repo's packages were not installed | Skills that learned "don't run tests", a rule about the laptop | 02, 03 |
| Cut the test set to 10 bugs | Differences under about 20 points could not be shown | 02, 03 |
| Tried to use Claude Code as a plain text pipe | Caught in a two-task trial: the model wrote pretend tool calls, which would have faked a large gain | 04 |
| Left a Windows encoding quirk in the practice logs | Caught before the test: the first skills carried a rule about it, so that step was redone | 04 |
- Protocol. Where the paper says how, we followed it: practice tasks run once, three skills written in three separate sessions by an unmodified Claude Code agent with the paper's instruction word for word, held-out tasks run three times with no skill and once with each skill, results given as mean and sample SD.
- Scoring. Pass or fail always comes from the benchmark's own tests. No model judges a result.
- Skill files are used exactly as written. Their hashes were recorded before the first test run.
- Differences from the paper are listed at the end of each report. The paper released no code and does not describe its harnesses, so none of these runs is an exact copy.
- The experiments were planned and run with Claude Code.
| Folder | Contents |
|---|---|
reports/ |
One report per experiment, with full tables |
skills/ |
Every skill file the skill writer produced, unchanged |
Raw logs and the runner scripts are not included yet.
- ALFWorld (the paper's largest claim): practice run done. Haiku 4.5 wins 86% of games with no skill, above what the paper's model reached with one. The test with skills has not been run.
- tau2-bench has not been run. It needs direct API access to the model.
- Next idea: a loop that tests each skill on the practice tasks and accepts it only if the pass rate does not drop.
- 2026-10-06: first version, covering the experiments of 2 to 4 October 2026.