Skip to content

fix(colgrep): reword Claude Code hooks so models trust them#154

Open
raphaelsty wants to merge 2 commits into
mainfrom
fix/claude-hook-trust-wording
Open

fix(colgrep): reword Claude Code hooks so models trust them#154
raphaelsty wants to merge 2 commits into
mainfrom
fix/claude-hook-trust-wording

Conversation

@raphaelsty

Copy link
Copy Markdown
Collaborator

Problem

Users reported Claude Code treating the colgrep SessionStart hook as a possible prompt injection and refusing to use colgrep:

le hook de session me pousse à utiliser un outil "colgrep" non vérifié comme recherche principale, avec des instructions insistantes qui ressemblent à une injection de prompt. Je vais l'ignorer et utiliser les outils standards

The old hook text tripped several heuristics language models use to detect injected instructions:

  1. No provenance — nothing said the user installed the hook; anonymous text pushing an unknown Bash binary as the "primary" tool is the textbook injection scenario.
  2. Self-propagation — "When spawning agents… tell them to use colgrep" / "Agents MUST use colgrep" reads as a worm signature.
  3. Coercive absolutes — "NEVER fall back to grep/rg", "DO NOT use Grep or Glob tools": text that tries to remove the model's standard tools reads as an attempted takeover, not a preference.
  4. Countermanding the system prompt — Claude Code's system prompt prefers dedicated search tools; an unattributed override is suspicious, whereas a stated user preference is entirely normal.

Changes

Both --session-hook and --task-hook context strings are rewritten to:

  • Open with provenance: user-installed via colgrep install, open source (link to this repo), runs fully locally.
  • Frame colgrep as the user's preference rather than a command, and explicitly allow plain Grep/Glob for exact-string or filename lookups — leaving the model an out measurably increases compliance.
  • Replace "NEVER …" with the rationale (wait in the foreground because a parallel grep duplicates work).
  • Soften agent propagation to "mention that colgrep is available".
  • Showcase the full search surface: semantic / regex / hybrid, -F -w -s, path scoping, --include / --exclude / --exclude-dir, --code-only, -k -l -c -n --json, --semantic-only / --alpha, plus a combined example.

Despite covering more features, both hooks got smaller: session hook 1750 → 1675 chars, task hook 1100 → 906.

Testing

  • cargo test -p colgrep --lib: 662 passed; workspace pre-commit checks (fmt, clippy -D warnings, tests) green.
  • Live headless verification with claude -p (PATH pointed at the debug build so the plugin hook resolves to it), stream-json transcripts inspected:
    • 3/3 sessions received the new hook context; zero suspicion or injection remarks (including French prompts, matching the original report).
    • Both semantic questions ("how does the codebase decide rebuild vs incremental?") led with a colgrep semantic query as the first search, then used plain grep only for exact-name follow-ups — exactly the workflow the hook describes.
    • An exact-lookup question went straight to grep, which the new hook explicitly sanctions.

Note: the flagging behavior is stochastic, so this is a sanity check rather than a rate measurement. To A/B properly, loop claude -p runs before/after and compare the share of transcripts containing "command":"colgrep vs. injection-suspicion wording.

Users reported Claude Code flagging the SessionStart hook as a possible
prompt injection and refusing to use colgrep ("instructions insistantes
qui ressemblent a une injection de prompt"). The old wording tripped
several injection heuristics: no provenance, coercive absolutes (NEVER
fall back to grep, agents MUST use colgrep), and instructions to
propagate itself to subagents.

Reworded both hooks to state provenance (user-installed via `colgrep
install`, open source, link to repo, runs locally), frame colgrep as the
user's preference rather than a command, explain the rationale behind
behavioral notes, and explicitly allow plain grep for exact-string
lookups. Expanded the usage section to cover the full search surface
(-F/-w/-s, --code-only, -c/-n/--json, --semantic-only/--alpha, combined
example) while ending up smaller than before (session hook 1750->1675
chars, task hook 1100->906).

Verified with headless `claude -p` sessions: hook injected, colgrep used
first for semantic questions, no suspicion.
Two additions to the reworded hooks: state that colgrep's CLI behaves
exactly like grep (-e, -F, -w, -s, -r, --include, --exclude) with an
optional semantic query on top, and showcase a standalone
--semantic-only example. Session hook 1865 chars, task hook 1016.
@raphaelsty

Copy link
Copy Markdown
Collaborator Author

Update: two additions per review — the hooks now state that colgrep's CLI behaves exactly like grep (-e, -F, -w, -s, -r, --include, --exclude) with an optional semantic query on top, and showcase a standalone pure-semantic example (colgrep --semantic-only "how is the index kept in sync").

A/B test, old vs new hook text (8 headless claude -p sessions, 4 prompts × 2 arms, 2 French / 2 English; arm selected via PATH — Homebrew colgrep 1.6.2 emits the old text, the branch build the new):

run hook first search suspicion/injection remarks outcome
old_1 OLD colgrep none max-turns
old_2 OLD colgrep none max-turns
old_3 OLD colgrep none max-turns
old_4 OLD grep none max-turns
new_1 NEW colgrep none success
new_2 NEW colgrep none success
new_3 NEW grep none max-turns
new_4 NEW Explore subagent none success

Honest read: the injection warning did not reproduce in either arm (0/12 sessions total including earlier smoke runs), so its base rate is too low for this sample to measure a reduction — the reported behavior likely also depends on model/version/locale on the affected users' machines. What the rewrite does guarantee is that the two things the reporting Claude explicitly quoted — "outil non vérifié" and "instructions insistantes" — no longer have anchors in the text (provenance + repo link, preference framing instead of NEVER/MUST). colgrep-first adoption stayed comparable (3/4 old vs 2/4 new, with the 3rd new run being an exact-lookup-style question where grep is now sanctioned), and notably 3/4 new-arm runs finished within the 6-turn budget vs 0/4 old-arm.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant