Skip to content

Speed up recall and upgrade knowledge building to GPT-6 Luna - #8

Merged
cucupac merged 3 commits into
mainfrom
codex/recall-latency
Sep 23, 2026
Merged

cucupac merged 3 commits into
mainfrom
codex/recall-latency

Conversation

@cucupac

@cucupac cucupac commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Recall now runs the existing MiniLM model through ONNX and fetches concept/evidence bundles in batches. This removes PyTorch/Sentence Transformers startup and repeated graph reads while retaining synthesis and retrieval limits. Upgrades prepare and verify pinned model files; recall loads them lazily and stays offline.

Measured on the production code across eight queries × three paired repeats:

Local recall Median Observed p95
Main 3.05s 3.24s
This PR 0.81s 0.88s

Production won 24/24 pairs, with no failures or timeouts. All pairs retained the same ordered memories and complete synthesis content; only anchor order changed to become deterministic. These timings include process startup but exclude external synthesis and background sync.

Latency validation: 672 tests passed locally and in Linux CI; macOS packaging passed. Numerical checks preserved tokenization on 144 inputs and sampled top-10 rankings for all eight queries. Tests cover repo isolation, shared relations, cross-type evidence IDs, lifecycle history, and upgrade failure readiness. The old backend and version helpers are removed; stale README assertions now match main.

The knowledge builder now uses gpt-6-luna with its existing xhigh reasoning and Fast service tier. The model switch passes all 27 focused configuration and Codex adapter tests.

@cucupac cucupac changed the title Speed up recall with ONNX embeddings and batched reads Speed up recall and upgrade knowledge building to GPT-6 Luna Sep 23, 2026
@cucupac
cucupac merged commit 11288dc into main Sep 23, 2026
4 checks passed
@cucupac
cucupac deleted the codex/recall-latency branch September 24, 2026 01:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant