Skip to content

feat(onnx): load-time warning when the model keys on integration-context fields - #103

Merged
omob merged 1 commit into
mainfrom
feat/context-sensitivity-probe
Jul 7, 2026
Merged

feat(onnx): load-time warning when the model keys on integration-context fields#103
omob merged 1 commit into
mainfrom
feat/context-sensitivity-probe

Conversation

@omob

@omob omob commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Follow-up to finding F3 from the efficacy validation (#99) — the detection half; the retrain that actually fixes the model is a separate piece of work.

What

A contextSensitivityProbe runs next to the existing calibration probe on every model load and registry hot-swap. It scores the same plausible transaction twice — once with the optional integration-context fields populated (is_authenticated, device_is_trusted, channel_code, currency_code, session_to_txn_seconds), once with them absent, exactly what an integrator sending only the six required fields produces. If the score gap exceeds 0.5 it logs a loud warning with remediation guidance; the measured gap is exposed via getModelInfo().

Warning only — never a readiness failure. The model still functions; the operator needs to know its scores track payload richness rather than behaviour.

Why

#99 measured the shipped model as a trust-context detector: 687/687 bare-payload transactions scored >0.65 while 770/770 full-context transactions scored ≤0.65 — mule networks, trusted-device velocity bursts, and rings all invisible. Nothing in the stack could see this failure mode. Now every load self-checks.

Verification

Run against the shipped fraud_model.onnx:

contextSensitivityProbe: Warn. Model score swings by 0.9998 on optional
integration-context fields alone (same transaction, context fields present
vs absent). bareScore=0.9998 fullScore=0.0000 gap=0.9998

Lint clean, existing onnx suite green.

🤖 Generated with Claude Code

…ext fields

New context-sensitivity probe alongside the calibration probe: score
the same plausible transaction with and without the optional
integration-context fields (is_authenticated, device_is_trusted,
channel, currency, session length). A gap above 0.5 logs a loud
warning and the gap is exposed via getModelInfo() — warning only,
never a readiness failure.

Against the shipped model the probe measures bare=0.9998 vs
full=0.0000 (gap 0.9998): the degeneracy from efficacy-validation
finding F3 is now self-detecting on every model load and hot-swap.

Found by efficacy-validation (finding F3, PR #99).
@omob
omob merged commit f0ae52c into main Jul 7, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant