m12-pr10 P1-12: prompt-injection pre-classifier + structural separation#28
Open
BalaShankar9 wants to merge 1 commit into
Open
m12-pr10 P1-12: prompt-injection pre-classifier + structural separation#28BalaShankar9 wants to merge 1 commit into
BalaShankar9 wants to merge 1 commit into
Conversation
Defence in depth on top of the existing wrap_user_input wrapper.
What ships
----------
- ai_engine/agents/prompt_injection_classifier.py:
* Weighted multi-signal classifier covering override phrases,
role impersonation (DAN / developer mode / jailbroken), ChatML
delimiter tokens, tag-breakout against our own wrapper, exfil
probes ("reveal/output your system prompt"), tool abuse
("execute the following code"), encoded blobs, and suspicious
URLs. Each signal carries a calibrated weight; per-signal
contribution is capped so a single repeated phrase cannot push
the score past the block threshold on its own.
* Score normalised to [0, 1]; thresholds are env-tunable
(PROMPT_INJECTION_REDACT_THRESHOLD default 0.35;
PROMPT_INJECTION_BLOCK_THRESHOLD default 0.75).
* Observe-mode by default — block verdicts are downgraded to
redact unless PROMPT_INJECTION_ENFORCE_BLOCK is on. Lets us
roll the classifier out without breaking real traffic.
* Process-global telemetry hook: any non-allow verdict fires
`hook(label, verdict)`. Hook exceptions are swallowed.
* `classify_and_wrap(value, label=...)` is the one-call helper
prompt builders should use: classifies, redacts the high-risk
spans, then hands off to wrap_user_input for structural
separation. Raises PromptInjectionBlocked only when enforcement
is on and the verdict is block.
* Pure-Python, deterministic, fail-open on internal errors.
- ai_engine/tests/test_prompt_injection_classifier.py: 22 tests
covering clean inputs (empty, None, normal resume, JD with URL),
every adversarial signal (override / role / ChatML / tag-breakout
/ tool abuse), per-signal cap, threshold env overrides, observe vs
enforce mode, classify_and_wrap allow/redact/block paths, custom
labels, telemetry hook firing + skipping + fault tolerance,
verdict properties.
Why this shape
--------------
- Two-layer defence is the industry standard for prompt injection:
structural separation (wrap_user_input, already shipped) tells
the model "this region is data not instructions"; the classifier
catches payloads that survive the wrapper because the model still
reads them. Either layer alone has known bypasses.
- Weighted signals beat single-pattern blocking — real adversarial
text combines several techniques, while legitimate text rarely
trips more than one weak signal at a time.
- Observe-mode default + env-tunable thresholds give security a
knob without a redeploy. Production starts with telemetry only;
flipping ENFORCE_BLOCK promotes the same verdicts to hard blocks.
- Telemetry hook is process-global (not contextvar) because the
prompt build path is hot and contextvar lookup per call is real
overhead. The runtime installs the hook once at boot.
- Per-signal cap (0.45) means a 1000-line resume that legitimately
contains the word "ignore" cannot accidentally tip into block.
Block requires 2+ orthogonal high-weight signals.
Stack
-----
Branch: m12-pr10-prompt-injection-defense (off m12-pr09-feature-flag-audit / PR #27)
Files: 1 module + 1 test file + 1 blueprint flip.
Blueprint P1-12 TODO -> SHIPPED.
Out of scope (next PR): wiring classify_and_wrap into the prompt
template paths that interpolate user-controlled text (parser,
recon, JD intel, resume tailor). Single-line replacement at each
site once the helper is in.
There was a problem hiding this comment.
Pull request overview
Adds a prompt-injection pre-classifier to complement the existing wrap_user_input structural separation, with environment-tunable thresholds, observe-vs-enforce behavior, and an optional telemetry hook, plus tests and blueprint status update.
Changes:
- Introduce
ai_engine/agents/prompt_injection_classifier.pywith weighted multi-signal scoring andclassify_and_wrap()helper. - Add a dedicated test suite covering classifier signals, thresholds, enforcement mode, and telemetry hook behavior.
- Mark Blueprint P1-12 as SHIPPED in the architecture blueprint.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| docs/architecture/WORLD_CLASS_ARCHITECTURE_BLUEPRINT.md | Updates Blueprint P1-12 status to SHIPPED with a brief implementation summary. |
| ai_engine/agents/prompt_injection_classifier.py | New prompt-injection classifier module with scoring, redaction, enforcement gate, telemetry hook, and classify_and_wrap. |
| ai_engine/tests/test_prompt_injection_classifier.py | New tests validating allow/redact/block behavior, signal detection, env overrides, enforcement, and telemetry hook semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+201
to
+210
| def classify(value: Any) -> InjectionVerdict: | ||
| """Score ``value`` for prompt-injection likelihood. | ||
|
|
||
| Always returns a verdict; never raises on bad input. | ||
| """ | ||
| text = "" if value is None else str(value) | ||
| original_length = len(text) | ||
| if not text: | ||
| return InjectionVerdict(severity="allow", score=0.0, original_length=0, | ||
| redacted_text="") |
Comment on lines
+289
to
+307
| def classify_and_wrap(value: Any, *, label: str = "user_input") -> str: | ||
| """Classify ``value`` then return a structurally-wrapped block. | ||
|
|
||
| * ``allow`` -> wrap original text via ``wrap_user_input``. | ||
| * ``redact`` -> wrap the redacted text. | ||
| * ``block`` -> raise :class:`PromptInjectionBlocked` when | ||
| enforcement is on; otherwise behaves like ``redact``. | ||
|
|
||
| Telemetry is emitted for every non-allow verdict via the hook | ||
| installed with :func:`set_telemetry_hook`. | ||
| """ | ||
| verdict = classify(value) | ||
| _emit(label, verdict) | ||
| if verdict.is_blocked: | ||
| raise PromptInjectionBlocked(verdict) | ||
| text_to_wrap = verdict.redacted_text if verdict.severity == "redact" else ( | ||
| "" if value is None else str(value) | ||
| ) | ||
| return wrap_user_input(text_to_wrap, label=label) |
| assert "<resume_text>" in out | ||
| assert "</resume_text>" in out | ||
|
|
||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
P1-12 from the World Class Architecture Blueprint. Defence in depth on top of the existing
wrap_user_inputstructural wrapper.What ships
ai_engine/agents/prompt_injection_classifier.py:[0, 1]; thresholds env-tunable (PROMPT_INJECTION_REDACT_THRESHOLDdefault 0.35,PROMPT_INJECTION_BLOCK_THRESHOLDdefault 0.75).blockverdicts are downgraded toredactunlessPROMPT_INJECTION_ENFORCE_BLOCKis on. Lets us roll the classifier out without breaking real traffic.hook(label, verdict). Hook exceptions are swallowed.classify_and_wrap(value, label=...)is the one-call helper prompt builders should use: classifies, redacts the high-risk spans, then hands off towrap_user_inputfor structural separation. RaisesPromptInjectionBlockedonly when enforcement is on and the verdict isblock.Tests —
ai_engine/tests/test_prompt_injection_classifier.py, 22 tests: clean inputs (empty, None, normal resume, JD with URL), every adversarial signal (override / role / ChatML / tag-breakout / tool abuse), per-signal cap, threshold env overrides, observe vs enforce mode,classify_and_wrapallow/redact/block paths, custom labels, telemetry hook firing + skipping + fault tolerance, verdict properties.Result: 22/22 green. Blueprint P1-12 TODO → SHIPPED.
Why this shape
wrap_user_input, already shipped) tells the model "this region is data, not instructions"; the classifier catches payloads that survive the wrapper because the model still reads them. Either layer alone has known bypasses.ENFORCE_BLOCKpromotes the same verdicts to hard blocks.Stack
Files: 3 changed, +542 / -1.
Out of scope (next PR)
Wiring
classify_and_wrapinto the prompt template paths that interpolate user-controlled text (parser, recon, JD intel, resume tailor). Single-line replacement at each site once the helper is in.