Skip to content

Commit 787f89c

Browse files
Improve opt-in Jev workflows, deadline guards, and evidence integrity (#252)
* feat: improve Jev workflows and streamline docs * fix: pin README assets for PR preview * fix: bound Jev review data and disclose remote recall * fix: preserve Jev support probabilities * fix: refresh registered offline benchmark evidence * docs: pin README links to refreshed evidence * fix: require explicit consent for remote Jev eval * docs: refresh offline benchmark evidence * docs: pin README links to refreshed chart * test: align contracts with pinned README links * fix: type Jev advisory planner path * docs: pin README to refreshed benchmark * test: accept refreshed benchmark image pin * fix: bind retrieval preview to grounded Jev plan * docs: pin README benchmark snapshot * docs: clarify managed Jev allowance terms * docs: align Jev plan details with Cloud policy * fix: screen Jev questions and correct recall fixture * docs: pin README chart to corrected evidence * fix: surface safe Jev fallback reasons * docs: pin Jev evidence to the updated snapshot * docs: clarify pooled Team Jev allowance * docs: pin pooled Jev plan guidance * docs: pin README to current Jev and benchmark evidence * fix(jev): preserve review evidence and recall compatibility * docs: pin README to refreshed benchmark evidence * Fix Jev ancestor review and positional API compatibility * fix: validate visible ancestor memories in Jev review * Pin corrected offline benchmark snapshot in every README consumer * docs: pin README to refreshed Jev review evidence * docs: preserve pre-merge Jev evidence snapshot * Enforce current Jev evidence visibility and serialized input bounds * Pin current temporal and byte-bound benchmark snapshot * fix: honor legacy secret classifications before Jev review * docs: pin Jev privacy evidence consumers to source snapshot * docs: pin README to benchmark v146 * fix: normalize Jev advisory fallback reasons for planning off * docs: pin normalized Jev advisory evidence snapshot * docs: pin README to benchmark v147 * docs: pin README to benchmark v148 * fix: update vulnerable benchmark PyJWT pin * docs: publish individual managed Jev limits * fix: address Jev review and docs CI failures * chore: export fresh offline evidence for Jev review * chore: capture immutable offline evidence outputs * chore: regenerate benchmark image exports * docs: register fresh offline evidence for Jev review * docs: checksum fresh offline evidence * docs: bind benchmark guide to refreshed artifact * test: bind evidence contract to refreshed artifact * docs: cite refreshed offline evidence * docs: refresh registered context evidence chart * docs: refresh registered grounded examples * chore: remove temporary offline evidence exporter * docs: pin benchmark links to refreshed evidence * fix(jev): keep local planning available and refresh offline evidence * docs: pin README to the verified offline evidence commit * docs: give planner evidence an unambiguous artifact identity * docs: pin the README to the distinct planner evidence * fix(pi): pin proxy-addr 2.0.8 for GHSA-jqcg-44mw-7w3h * fix(mcp): reject managed custom decisions before credential access * docs: pin guides and visuals to the managed preflight fix * docs: pin benchmark links to verified PR252 evidence * fix(jev): preserve offline review and enforce advisory deadlines Restore canonical v149 integrity, guard Latest promotion after uploads, and bind fresh immutable offline evidence. Preserve external Ask/UI fixes and qualify offline provider and workspace fixtures. * docs: pin benchmark evidence to the verified source checkpoint Use the actual connector-created source commit for the benchmark guide and figure links. Fifty affected evidence and documentation checks pass; the runtime tree is unchanged from the qualified offline repair.
1 parent f0f0303 commit 787f89c

110 files changed

Lines changed: 18061 additions & 1788 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude-plugin/skill-assets.sha256‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,4 @@ c5d0c26f28c9ee14092f9deaf24c98dd8bef49d971fef2b7a537ffb1ab9f2887 .claude-plugin
33
aeee7a94671ceb306fe2d24c5acc9f2d96ad8a8e7410536566799eea6265f080 skills/engraphis-memory/SKILL.md
44
055655db84af07561d002f0c69744313d8413c39f3e873f941f0fa0b1e76dc66 skills/engraphis-memory/references/CONVENTIONS.md
55
9d090a03f5b3f36a34d91f66b72c3844591f6915755ac3a6c6ba5f1b16977de5 skills/engraphis-memory/references/SCOPING.md
6-
68cd0add0d1025d495da2163e7c462b76b5fd1a4ff86b527fee49a4d8d6ab29b skills/engraphis-memory/references/TOOLS.md
6+
db1db0df676e4c05fd80b878a1bd1ffc9f2045c0b0e01c0486b69ce0a5541b73 skills/engraphis-memory/references/TOOLS.md

‎.github/workflows/release.yml‎

Lines changed: 17 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1251,12 +1251,8 @@ jobs:
12511251
notes_args=()
12521252
latest_args=(--latest)
12531253
promote_latest=true
1254-
if [ "$WAIVE_QUALIFICATION" = "true" ]; then
1255-
# A retained older repair must not displace a newer public Latest.
1256-
# The candidate comparison uses the existing public release history;
1257-
# failed lookups or unknown tag formats stop before any release write.
1258-
current_latest="$(gh release view --repo "$GH_REPO" --json tagName --jq .tagName)"
1259-
promote_latest="$(python - "$RELEASE_TAG" "$current_latest" <<'PY'
1254+
can_promote_latest() {
1255+
python - "$RELEASE_TAG" "$1" <<'PY'
12601256
import re
12611257
import sys
12621258
@@ -1267,7 +1263,13 @@ jobs:
12671263
12681264
print("true" if version(sys.argv[1]) >= version(sys.argv[2]) else "false")
12691265
PY
1270-
)"
1266+
}
1267+
if [ "$WAIVE_QUALIFICATION" = "true" ]; then
1268+
# A retained older repair must not displace a newer public Latest.
1269+
# The candidate comparison uses the existing public release history;
1270+
# failed lookups or unknown tag formats stop before any release write.
1271+
current_latest="$(gh release view --repo "$GH_REPO" --json tagName --jq .tagName)"
1272+
promote_latest="$(can_promote_latest "$current_latest")"
12711273
latest_args=(--latest=false)
12721274
if [ "$promote_latest" = "true" ]; then latest_args=(--latest); fi
12731275
{
@@ -1293,7 +1295,14 @@ jobs:
12931295
--repo "$GH_REPO" \
12941296
--clobber
12951297
if [ "$WAIVE_QUALIFICATION" = "true" ] && [ "$promote_latest" = "true" ]; then
1296-
gh release edit "$RELEASE_TAG" --repo "$GH_REPO" --latest
1298+
# Another publisher may have advanced Latest during asset upload.
1299+
# Recheck immediately before promotion; this narrows the race but
1300+
# does not provide an atomic lease across external publishers.
1301+
current_latest="$(gh release view --repo "$GH_REPO" --json tagName --jq .tagName)"
1302+
promote_latest="$(can_promote_latest "$current_latest")"
1303+
if [ "$promote_latest" = "true" ]; then
1304+
gh release edit "$RELEASE_TAG" --repo "$GH_REPO" --latest
1305+
fi
12971306
fi
12981307
else
12991308
gh release create "$RELEASE_TAG" verified-dist/* release-evidence/* \

‎BENCHMARKS.md‎

Lines changed: 39 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -93,15 +93,15 @@ interpretation and do not count as additional benchmark-quality gains.
9393

9494
### Public numeric evidence registry
9595

96-
Every exact public aggregate retained below comes from the checked-in, public-safe
97-
[`offline-fixtures-v129.json`](docs/benchmark-evidence/offline-fixtures-v129.json) artifact. Its
98-
SHA-256 is
99-
`686394aa5d69cd296360defc5170d956c85565a5282c7766dc6466ba4c6aa6a5`, also recorded in the
96+
Every exact public aggregate retained below comes from the checked-in, public-safe
97+
[`offline-fixtures-pr252-four-gap-guards-20261007.json`](docs/benchmark-evidence/offline-fixtures-pr252-four-gap-guards-20261007.json) artifact. Its
98+
SHA-256 is
99+
`c53b82e4a847bf525c695c25a1813e8a591e86e39ca9e652e385e898ecd04010`, also recorded in the
100100
adjacent `.sha256` file. The artifact contains no raw questions, answers, prompts, customer data,
101101
or per-record content fingerprints.
102102

103-
The fixture-suite digest is
104-
`8e2a562ba5d4c785c1e88481f9019446b43608109147cd7cd82597493004af35`. The artifact defines
103+
The fixture-suite digest is
104+
`e137eacdd70bbea7539dec6ed5cd4b59c980352a8e641094b449e474522aa109`. The artifact defines
105105
the digest algorithm and records the SHA-256 of every suite and dataset file. Each evidence ID
106106
also binds its exact command through `sha256(UTF-8 exact command)`:
107107

@@ -121,12 +121,19 @@ means no number is claimed in this offline registry.
121121
The context-efficiency chart is generated from the registry values and the selected report schema.
122122
Historical LoCoMo, graph, handoff, consolidation, and security figures remain preserved in their
123123
source artifacts but are omitted from the current chart until each has a matching immutable,
124-
public-safe artifact. The chart labels coding outcomes, external datasets, and operational
125-
capacity as pending evaluation tracks rather than implying scores. Regenerate it with
126-
`python scripts/render_benchmark_report.py --report docs/benchmark-evidence/offline-fixtures-v129.json --output docs/images/context-efficiency.svg` after selecting the report to publish.
124+
public-safe artifact. The chart reports context size, candidate and packed retrieval quality, and
125+
grounded checks in separate panels; provider billing and MCP transport are not measured. Regenerate
126+
the SVG and matching PNG with:
127+
128+
```bash
129+
python scripts/render_benchmark_report.py --report docs/benchmark-evidence/offline-fixtures-pr252-four-gap-guards-20261007.json --output docs/images/context-efficiency.svg --png-output docs/images/context-efficiency.png
130+
```
127131

128-
The companion examples are also generated from that artifact with
129-
`python -m scripts.render_benchmark_examples --report docs/benchmark-evidence/offline-fixtures-v129.json --output docs/images/evidence-backed-agent-examples.svg`.
132+
The companion examples are also generated from that artifact with:
133+
134+
```bash
135+
python -m scripts.render_benchmark_examples --report docs/benchmark-evidence/offline-fixtures-pr252-four-gap-guards-20261007.json --output docs/images/evidence-backed-agent-examples.svg --png-output docs/images/evidence-backed-agent-examples.png
136+
```
130137
The historical-to-executable mapping is in
131138
[`docs/BENCHMARK_CHANGE_COVERAGE.md`](docs/BENCHMARK_CHANGE_COVERAGE.md).
132139

@@ -290,6 +297,27 @@ LoCoMo and LongMemEval retrieval diagnostics are retained as separate public-saf
290297
and checksum boundaries. Those values remain evidence-retrieval metrics, not end-to-end QA accuracy
291298
or an official LoCoMo leaderboard score.
292299

300+
## Jev-assisted route-selection probe (provider-backed, exploratory)
301+
302+
On 2026-10-03, the configured BYOK client ran `eval/jev_recall_quality.py` against 40 generated
303+
public synthetic tasks (10 each for lexical identifiers, graph relationships, temporal changes,
304+
and procedural context). Each call sent only the task query and at most two locally generated
305+
routes; no recalled memory bodies or stored user data were sent. Jev selected a route on all 40
306+
calls, but the baseline and Jev-assisted means were identical: nDCG@5 **0.740773**, Recall@5
307+
**0.75**, and answer-token coverage **0.75**. Every paired 95% interval was zero, so the
308+
improvement/non-inferiority gate did not pass. This synthetic probe does not establish a benefit
309+
on held-out user workloads, and no retrieval-quality improvement is claimed.
310+
311+
The harness caps a run at 40 provider requests, makes at most one request per task, and requires
312+
explicit `--allow-remote` consent. Repeating it can consume BYOK provider usage:
313+
314+
```bash
315+
python -m eval.jev_recall_quality --allow-remote --max-requests 40 --timeout 8
316+
```
317+
318+
This exploratory provider run is separate from the registered offline artifact and the benchmark
319+
graphic below; it is not a general benchmark or customer-workload evaluation.
320+
293321
## What we do NOT yet claim
294322

295323
- **No official end-to-end LLM QA accuracy.** The deterministic productivity agent measures the

0 commit comments

Comments
 (0)