Make scoped memory writes, retrieval and workspace processing reliable (#201)

* feat(core): deterministic context packer, NumPy PageRank, recall tuning, gist format, schema indexes

- Add DeterministicContextPacker with inter-candidate clause redundancy pruning,
  score-elbow gating, and ContextPackResult NamedTuple API
- Vectorize personalized_pagerank with NumPy (np.add.at scatter, vectorized
  dangling mass, L1 convergence) for graph performance on large stores
- Add early-return guard when incidence_memory_ids is empty in recall pipeline
- Reduce list_memory_ids candidate limit 12000 to 500 for lower latency
- Add 4 schema indexes: entities workspace_created, edge dst_visibility,
  mem_links a_valid and b_valid for graph/link traversal performance
- Add format='gist' to engraphis_recall_context and smart_recall_context
  for 60-80% token savings with one-line memory summaries
- Add diagnostics pruning: strip verbose default-valued fields when
  diagnostics=False for cleaner MCP responses
- Export ContextPackResult, DeterministicContextPacker, pack_context from
  engraphis.core public API
- Add 29 tests for context packer and 2 tests for MCP gist/diagnostics

* fix(store,docs): defer mem_links temporal index creation and document format param in skills

- Defer idx_mem_links_b_valid creation in Store._apply_schema alongside idx_mem_links_temporal to preserve legacy v5 migration compatibility
- Document format parameter for engraphis_recall_context in portable skill reference and MCP_TOOLS.md
- Refresh .claude-plugin/skill-assets.sha256 digest

* docs(changelog),tools: refine disclosure prose, document config path, and add multi-mode slider test tool

- Direct cross-encoder reranker config to ~/.engraphis/config.env rather than CWD .env
- Soften security disclosure phrasing in CHANGELOG.md
- Add multi-mode slider regression harness tools/galaxy_mode_test.js

* docs(changelog): replace em-dash with semicolon to satisfy docs punctuation policy

* feat: strengthen memory correctness and workspace processing controls

* docs: record reliability review and validation evidence

* test: isolate offline doctor and optional HTTP policy coverage

* fix: address review feedback on repair queue indexing, editable extras, and MCP contract normalization

* fix: preserve evidence and repair recreated external indexes

* fix: enforce operator veto and synchronize physics tests

Author: Coding-Dev-Tools

.claude-plugin/skill-assets.sha256

@@ -3,4 +3,4 @@ a8307092284d9ab4ba62f4f089d14a674c33d0f241a8430ffd89abb19e5f1ca0  .claude-plugin
 4bc8979b9ffeb97190960e551dbf4ddc6f7aeeb7b86894fd2298a59ff0001efa  skills/engraphis-memory/SKILL.md
 055655db84af07561d002f0c69744313d8413c39f3e873f941f0fa0b1e76dc66  skills/engraphis-memory/references/CONVENTIONS.md
 62019760766ff472a76a0f81437898f39e3c1fe2631732b7b7733e50c1ad837f  skills/engraphis-memory/references/SCOPING.md
-d65721c1cc29faf975138a99f07bdee29ac49a9bc64b0737001781abc14407be  skills/engraphis-memory/references/TOOLS.md
+33874c7c7a1c0911b0e73c7d22addc9828963d5436cb315fe7c6c5587c6b911d  skills/engraphis-memory/references/TOOLS.md

.github/workflows/ci.yml

@@ -256,11 +256,19 @@ jobs:
           python -m pip install --upgrade pip
           pip install -e ".[test]" "uvicorn[standard]>=0.29"
           npm ci --ignore-scripts --omit=optional
-          npx playwright install --with-deps chromium
+          npx playwright install --with-deps chromium firefox webkit
       - name: Audit the root browser dependency lock
         run: npm audit --audit-level=high
       - name: Playwright desktop/mobile, keyboard, CSP, console, and axe checks
         run: npx playwright test
+      - name: Retain browser failure diagnostics
+        if: failure()
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
+        with:
+          name: browser-failure-diagnostics-${{ github.run_id }}-${{ github.run_attempt }}
+          path: test-results/
+          if-no-files-found: ignore
+          retention-days: 14
 
   docker-gate:
     # Keeps CI fast: the docker job always runs on push to main, but on PRs only

AGENTS.md

@@ -29,7 +29,7 @@ most common mistake here.
 | Status | Primary scoped, bi-temporal, interface-driven implementation. | Compatibility/reference implementation with flat namespaces. |
 | Model | Scoped + bi-temporal + typed; interface-driven. | Single flat `namespace` string per memory. |
 | Code | `engraphis/core/`, `engraphis/backends/`, `eval/`, `tests/`, `scripts/migrate_to_v2.py` | `engraphis/app.py`, `config.py`, `models.py`, `routes/`, `stores/`, `engines/`, `llm/`, `static/` |
-| Data | new v2 schema (`SCHEMA_VERSION = 16`) | `engraphis_v1.db` |
+| Data | new v2 schema (`SCHEMA_VERSION = 17`) | `engraphis_v1.db` |
 | Entry | `engraphis.MemoryEngine.create()` / `engraphis.create_memory_engine()` → `engraphis/factory.py` → `core/engine.py` | Internal reference only; never a public launcher |
 
 **Rule:** build new capability on **v2** (`core/` + `backends/`) behind the interfaces.
@@ -210,7 +210,7 @@ These are pure, unit-tested functions — change them only with a corresponding
 
 ---
 
-## 5. Data model cheat-sheet (`core/interfaces.py`, `core/schema.py` — `SCHEMA_VERSION = 16`)
+## 5. Data model cheat-sheet (`core/interfaces.py`, `core/schema.py` — `SCHEMA_VERSION = 17`)
 
 - **Scope hierarchy:** `workspace → repo → session → memory`. Scopes: `session|repo|workspace|user`.
 - **Bi-temporal validity on every record:** world-time `valid_from/valid_to` +

CHANGELOG.md

@@ -3,6 +3,28 @@
 All notable changes to Engraphis are documented here. Format loosely follows
 [Keep a Changelog](https://keepachangelog.com/); versions use SemVer.
 
+## [Unreleased]
+
+### Reliability and privacy
+
+- Preserve distinct context claims, qualified sentences and complete units under tight budgets;
+  measure false NOOP outcomes through real write sequences.
+- Preserve separate sources during packing and keep MCP gist responses within the canonical
+  context budget. Response caps retain or omit complete context and report accurate usage.
+- Canonical temporal browsing, server-side Library filtering/pagination, independent Ask states,
+  actionable setup diagnostics and retained installation capabilities.
+- Cross-process write resolution and schema 17 durable vector-index repair, with canonical
+  fallback and bounded NumPy scans. Public engine entrypoints remain compatible.
+- Commit native batch indexing with canonical memory state and roll back both on failure.
+  Retain the established 12,000-memory graph window pending quality evidence for a smaller one.
+- Explicit workspace managed-processing approval; missing legacy policy pauses readable uploads.
+  Requires the compatible cloud migration before rollout. Encrypted sync remains separate.
+- Generated Smart/Classic MCP contract and integration inputs; Pro three-day and Team ten-day
+  trial copy aligned with cloud authority. Real browser and Workers evidence remains distinct
+  from production verification. See `docs/RELIABILITY_PROGRAM.md`.
+- Isolate the manual graph diagnostic on an available local port with a private in-memory
+  server; fail before contacting an existing service when the requested port is occupied.
+
 ## [1.7.1] - 2026-09-03
 
 ### Fixed
@@ -36,9 +58,8 @@ All notable changes to Engraphis are documented here. Format loosely follows
   (savings_ratio 0.0 -> 0.4975) with no caller-side arguments. The packer is the
   existing 1.6 contract; the change just makes it the default fast path.
 - Smart MCP `engraphis_remember` now accepts and forwards `subject_key` and
-  `claim_kind` to the classic tool. Without this, every keyed write silently stored
-  empty keys because the served gateway surface dropped the parameters; the
-  documented safe-supersession mechanism is now reachable through MCP.
+  `claim_kind` to the classic tool, so the documented safe-supersession
+  mechanism is reachable through MCP.
 - A new integration at `integrations/commandcode/session_start_hook.py` (with
   `scripts/install_cc_hook.py` for idempotent user-scope install/uninstall) wires
   durable-memory recall into Command Code's SessionStart lifecycle: each new
@@ -50,7 +71,11 @@ All notable changes to Engraphis are documented here. Format loosely follows
   / `ENGRAPHIS_RERANK_MODEL`). Evaluated offline on the bundled retrieval gates
   (sample.jsonl, codemem.jsonl, k=5): hit@5 stays at 1.0 with zero per-question
   regressions, MRR@5 lifts 0.889 -> 0.944 (sample) and 0.962 -> 0.981 (codemem),
-  with ~15 ms per query added. Not the default; flip with a one-line config.
+  with ~15 ms per query added. Not the default; set the value in the trusted
+  config file (`~/.engraphis/config.env` on the operator account, or as a
+  process environment variable); Engraphis deliberately does not read the
+  CWD `.env`, so editing `./.env` and restarting leaves the identity
+  reranker active. Restart the MCP server and dashboard after the change.
 
 ### Changed
 
@@ -78,10 +103,8 @@ All notable changes to Engraphis are documented here. Format loosely follows
 
 ### Fixed
 
-- The Smart MCP gateway `engraphis_remember` binding was silently dropping
-  `subject_key` and `claim_kind`; this is the underlying cause of the
-  benchmark correction-miss pattern that the reworded-correction detector
-  then had to compensate for.
+- The Smart MCP gateway `engraphis_remember` now forwards `subject_key` and
+  `claim_kind` end to end, matching the **Added** entry above.
 
 ### Operational
 

README.md

@@ -804,7 +804,7 @@ file. It never searches the working directory for `.env`, and explicit process v
 | `ENGRAPHIS_CLOUD_REFRESH_CREDENTIAL` | Not set | Bootstrap-only rotating hosted credential; after first use the owner-only cloud session replacement takes precedence |
 | `ENGRAPHIS_CLOUD_TOKEN_SUBJECT` | `member` | Subject fixed during hosted bootstrap (`device` or `member`); set explicitly with an environment-only refresh credential |
 | `ENGRAPHIS_CLOUD_ACCESS_TOKEN` | Not set | Optional short-lived access token for ephemeral jobs |
-| `ENGRAPHIS_MANAGED_COMPUTE_CONSENT` | *(auto)* | Operator override only; default follows whether a cloud session is configured (connected = allowed, local-only = never). `0` opts a connected installation out; `1` permits local snapshot preparation but does not create a cloud credential or authorize an upload |
+| `ENGRAPHIS_MANAGED_COMPUTE_CONSENT` | *(unset)* | Deny-only operator override: `0` pauses readable managed processing. A truthy value cannot grant approval. Each workspace requires explicit confirmation in Manage → Settings; encrypted sync is separate |
 
 The optional cross-encoder reranker is model- and hardware-dependent. Treat its quality and
 latency as deployment-specific until a versioned model identity, exact configuration, and
@@ -869,3 +869,16 @@ under Apache-2.0 keeps that grant; later releases cannot retroactively withdraw
 official hosted control plane, its production credentials and records, managed operations,
 support, and future separately delivered commercial modules are outside the public source
 grant. See [`docs/LICENSING.md`](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/LICENSING.md) for the complete boundary.
+
+### Reliability implementation candidate
+
+The current source uses schema 17 for durable, content-free vector-index repair.
+See [the reliability program](https://github.com/Coding-Dev-Tools/engraphis/blob/main/docs/RELIABILITY_PROGRAM.md) for exact implementation,
+validation, migration and release boundaries. Managed processing now requires explicit
+workspace approval in Manage → Settings. Existing installations start with readable
+uploads paused until confirmed; connecting an account does not grant approval.
+
+For setup diagnostics use `engraphis-init --check --json`. New configurations get an
+owner-private local API token. Existing configs are preserved. Record selected install
+capabilities with `engraphis-init --extras server,mcp` or `--extras none`; future updates
+preserve that choice. `ENGRAPHIS_UPDATE_EXTRAS` remains an explicit override.

docs/HOSTED_PLANS.md

@@ -23,7 +23,7 @@ implementations are not part of this repository.
 
 Start or manage a hosted subscription in the [Engraphis account portal](https://api.engraphis.com/account?plan=pro&interval=monthly&utm_source=engraphis&utm_medium=docs&utm_campaign=pro_conversion&utm_content=hosted_plans_pricing#billing).
 
-The email-confirmed, no-card trial lasts three active days. If hosted entitlement expires,
+The email-confirmed, no-card trial lasts three active days for Pro and ten active days for Team. If hosted entitlement expires,
 `workspace_write_grace` can retain only approved hosted-account continuity operations for up to
 24 hours. It does not extend a trial or subscription, grant cloud access, or affect the free
 local tools. `recovery_read_only` supports hosted account recovery and export after grace.

docs/HOSTING_RAILWAY.md

@@ -50,8 +50,8 @@ Prefer mounting the owner-only cloud session file rather than placing a rotating
 credential directly in deployment configuration. An injected environment credential is only the
 bootstrap value; after rotation, the owner-only saved replacement takes precedence. **Cloud Sync
 encrypts eligible shared-workspace changes end-to-end before they leave the device; Engraphis
-Cloud cannot read their contents.** Managed compute is separate: once connected, it is enabled by
-default for an authorized customer and may upload a readable snapshot capped at 16 MiB over HTTPS
+Cloud cannot read their contents.** Managed compute is separate: every workspace must be
+explicitly approved in Manage → Settings before a readable snapshot capped at 16 MiB may upload over HTTPS
 to produce results. Secret-class and session-scoped rows are excluded client-side, and
 secret-class rows are rejected server-side.
 Set `ENGRAPHIS_MANAGED_COMPUTE_CONSENT=0` to opt the deployed installation back out.