Check tool results for patient data before they leave the machine - #89
Conversation
A tool result does not stay local. It goes back to the MCP client, which puts it into the model's context, and when that client is Claude Desktop or any other hosted model the result has just been sent to a third party. So the risky direction for this server is outbound: a tool that reads the MoleCare API can return a real person's clinical narrative, and no pattern scanner sees it, because it is prose in a tool result rather than a marker in a prompt. registerTools has a single choke point every result passes through, so the check goes there. After dispatch, the text blocks of the result are sent to a local privacy gate (MoleCare/privacy-gate-llm: an embedding model and a 1024-weight head). If it holds, the tool returns PRIVACY_HOLD instead of the data, and the log records the score, never the text. Off unless PRIVACY_GATE_URL is set, which is every published install. This server works with no credentials and no services by design, and a privacy feature must not change that. It reaches the gate over HTTP, so the package gains no model weights and no dependency. Public reference tools are exempt: SNOMED and ICD-10 lookups, ABCDE education, risk-factor prompts. They name no person, and gating them would add an embedding call per lookup for nothing. The allowlist is the exemption rather than the rule on purpose, so a tool added later is checked by default and someone has to think before exempting it. It fails closed. An unreachable gate, a 503, or a malformed answer all hold, because a gate that cannot answer must not be the reason a record reached a hosted model. The cost of being wrong that way is a refused call the user can see. Tests: 79, including the gate's 16. No network; fetch is injected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Greptile SummaryThe PR adds an optional HTTP privacy gate to the shared MCP tool-result path.
Confidence Score: 3/5The PR is not yet safe to merge because malformed or unsuccessful privacy-gate responses can release the patient data the feature is intended to withhold. Gate responses are cast rather than runtime-validated, allowing missing or invalid verdicts to become authoritative pass decisions before the original tool result is returned. Files Needing Attention: src/privacy-gate.ts
|
| Filename | Overview |
|---|---|
| src/privacy-gate.ts | Introduces privacy-gate configuration, allowlisting, text extraction, and HTTP verdict handling, but incomplete or unsuccessful responses can fail open. |
| src/runtime.ts | Integrates the privacy check into the common post-dispatch result path and substitutes held results with a safe error. |
| tests/privacy-gate.test.mjs | Adds broad unit coverage, though it omits successful JSON without hold and non-OK responses containing hold: false. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Tool call] --> B[Validate and dispatch]
B --> C[Tool result]
C --> D{Gate configured and tool checked?}
D -- No --> E[Return original result]
D -- Yes --> F[Extract text blocks]
F --> G[POST to local privacy gate]
G --> H{hold is true?}
H -- Yes --> I[Return PRIVACY_HOLD]
H -- No --> E
Reviews (1): Last reviewed commit: "Check tool results for patient data befo..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| if (!response.ok) { | ||
| // The sidecar answers 503 with hold:true when its embedding endpoint is | ||
| // down. Honour whatever it said, and hold if it said nothing useful. | ||
| return { hold: body?.hold !== false, score: null, unavailable: true }; | ||
| } | ||
| return { hold: Boolean(body.hold), score: body.score ?? null, unavailable: false }; |
There was a problem hiding this comment.
When the configured gate returns a successful but incomplete response such as {}, Boolean(body.hold) becomes false. A failed response containing hold: false also passes. In either case, the original checked patient result is returned to the client, which breaks the promised fail-closed behavior and can expose clinical data.
How this was verified: The unchecked verdict is authoritative in
runtime.ts, which returns the original tool result whenever these branches producehold: false.
| if (!response.ok) { | |
| // The sidecar answers 503 with hold:true when its embedding endpoint is | |
| // down. Honour whatever it said, and hold if it said nothing useful. | |
| return { hold: body?.hold !== false, score: null, unavailable: true }; | |
| } | |
| return { hold: Boolean(body.hold), score: body.score ?? null, unavailable: false }; | |
| if (!response.ok || typeof body?.hold !== "boolean") { | |
| return { hold: true, score: null, unavailable: true }; | |
| } | |
| return { hold: body.hold, score: body.score ?? null, unavailable: false }; |
|
Thanks for merging the check for tool results before patient data leaves the machine, @{@YauhenBichel}!
|
1 participant
Wires privacy-gate-llm into the one place every tool result passes. Off unless
PRIVACY_GATE_URLis set — every published install behaves exactly as today.Why the outbound direction
A tool result goes back to the MCP client and into the model's context. With Claude Desktop or any hosted model, that result has just left the machine. So
get_user_moles,get_mole_analysisand the rest can hand a real person's clinical narrative to a third party — and no pattern scanner catches it, because it's prose in a tool result, not a marker in a prompt.What it does
In
registerTools, afterdispatch, the result's text blocks go to a local gate. If it holds, the tool returnsPRIVACY_HOLDinstead of the data. The log records the score, never the text.PRIVACY_GATE_URL, no calls. This server works with no credentials and no services by design.scripts/serve.pyin privacy-gate-llm).What it deliberately does not do
It does not expose the gate as an MCP tool. For inbound text that protects nothing — by the time the model can call a tool with the text, it's already in context. The check has to sit in the result path, which is where this puts it.
Tests
79 pass, including 16 new ones, no network (
fetchis injected). They cover: off means no calls, fails closed on every error shape, user-data tools checked, reference tools exempt, and an unknown future tool checked by default.Separate PR coming
While building this I found
node_modulescommitted as a self-referential absolute symlink (added in #76). It breaks fresh checkouts —tscsilently does nothing — and leaks a home path. It's unrelated to this change, so it gets its own PR rather than riding along here.