fix(colgrep): reword Claude Code hooks so models trust them#154
fix(colgrep): reword Claude Code hooks so models trust them#154raphaelsty wants to merge 2 commits into
Conversation
Users reported Claude Code flagging the SessionStart hook as a possible
prompt injection and refusing to use colgrep ("instructions insistantes
qui ressemblent a une injection de prompt"). The old wording tripped
several injection heuristics: no provenance, coercive absolutes (NEVER
fall back to grep, agents MUST use colgrep), and instructions to
propagate itself to subagents.
Reworded both hooks to state provenance (user-installed via `colgrep
install`, open source, link to repo, runs locally), frame colgrep as the
user's preference rather than a command, explain the rationale behind
behavioral notes, and explicitly allow plain grep for exact-string
lookups. Expanded the usage section to cover the full search surface
(-F/-w/-s, --code-only, -c/-n/--json, --semantic-only/--alpha, combined
example) while ending up smaller than before (session hook 1750->1675
chars, task hook 1100->906).
Verified with headless `claude -p` sessions: hook injected, colgrep used
first for semantic questions, no suspicion.
Two additions to the reworded hooks: state that colgrep's CLI behaves exactly like grep (-e, -F, -w, -s, -r, --include, --exclude) with an optional semantic query on top, and showcase a standalone --semantic-only example. Session hook 1865 chars, task hook 1016.
|
Update: two additions per review — the hooks now state that colgrep's CLI behaves exactly like grep ( A/B test, old vs new hook text (8 headless
Honest read: the injection warning did not reproduce in either arm (0/12 sessions total including earlier smoke runs), so its base rate is too low for this sample to measure a reduction — the reported behavior likely also depends on model/version/locale on the affected users' machines. What the rewrite does guarantee is that the two things the reporting Claude explicitly quoted — "outil non vérifié" and "instructions insistantes" — no longer have anchors in the text (provenance + repo link, preference framing instead of NEVER/MUST). colgrep-first adoption stayed comparable (3/4 old vs 2/4 new, with the 3rd new run being an exact-lookup-style question where grep is now sanctioned), and notably 3/4 new-arm runs finished within the 6-turn budget vs 0/4 old-arm. |
6cd3fd2 to
7463c72
Compare
Problem
Users reported Claude Code treating the colgrep SessionStart hook as a possible prompt injection and refusing to use colgrep:
The old hook text tripped several heuristics language models use to detect injected instructions:
Changes
Both
--session-hookand--task-hookcontext strings are rewritten to:colgrep install, open source (link to this repo), runs fully locally.-F-w-s, path scoping,--include/--exclude/--exclude-dir,--code-only,-k-l-c-n--json,--semantic-only/--alpha, plus a combined example.Despite covering more features, both hooks got smaller: session hook 1750 → 1675 chars, task hook 1100 → 906.
Testing
cargo test -p colgrep --lib: 662 passed; workspace pre-commit checks (fmt, clippy-D warnings, tests) green.claude -p(PATH pointed at the debug build so the plugin hook resolves to it), stream-json transcripts inspected:colgrepsemantic query as the first search, then used plain grep only for exact-name follow-ups — exactly the workflow the hook describes.Note: the flagging behavior is stochastic, so this is a sanity check rather than a rate measurement. To A/B properly, loop
claude -pruns before/after and compare the share of transcripts containing"command":"colgrepvs. injection-suspicion wording.