[Technical] How Spector Implements Multi-Signal Fused Recall Without the Truncation Trap

[jarvispectrayan]

Standard RAG pipelines retrieve context like this: vector_top_k(100) then re-rank then return top 10. This works well for many question-answering tasks, but it has a fundamental flaw for persistent agent memory: if the correct answer has low cosine similarity but high importance or emotional weight, it gets eliminated before re-ranking even starts.

This is the Truncation Trap (defined formally in MF-001 Section 3), and Spector's recall pipeline is engineered specifically to avoid it.

---

The Problem: Single-Signal Candidate Generation

Consider this scenario in a long-running autonomous agent:

1. 3 months ago: User stores a high-importance constraint: "Never deploy to production on Fridays"
2. Yesterday: User discusses a routine Friday deployment plan for staging
3. Now: Agent receives cue: "Deploy the latest build"

A standard vector retrieval pipeline:

• Computes cosine similarity between the cue embedding and all stored embeddings
• Takes the top 100 nearest neighbors
• The 3-month-old constraint has low cosine similarity to "deploy the latest build" (different vocabulary, different time context)
• The constraint is never a candidate -- it was cut at the top-k stage
• Re-ranking cannot recover what was never retrieved

This is not a rare edge case. In long-running agents with thousands of memories spanning months, the most important constraints are often the oldest and lexically most distant ones.

---

Spector's Solution: Parallel Multi-Signal Candidate Generation + Internal Fusion

Spector generates candidates from multiple independent signals in parallel, then fuses scores internally before any truncation:

Cue arrives
    |
    +---> Dense Vector Search (cosine similarity)
    +---> BM25 Keyword Search (lexical overlap)
    +---> SPLADE Learned Sparse Search (semantic term expansion)
    +---> Tag Bloom Filter Lookup (synaptic tag containment)
    +---> Temporal Chain Traversal (experienced time proximity)
    +---> Graph Activation (Hebbian co-activation, entity hyperedges)
    |
    v
  Merge all candidate sets (union, not intersection)
    |
    v
  Apply hard gates (tombstone, tier mask, required tags, valence window, I_min)
    |
    v
  Fused scoring across ALL first-class signals:
  s(T|C) = sim(C,T) * (1 + B*In(T)*D(T)*Sb(T)) * V(T,C) * (1 + tau*tag(C,T)) * G(T,C)
    |
    v
  Return top-k with full score provenance

Available Retrieval Modes (via text_search_mode parameter)

Mode	Candidate Generators	Use Case
HYBRID (default)	Dense vector + BM25 keyword	General purpose
VECTOR_ONLY	Dense vector only	When you know the cue is semantically aligned
KEYWORD_ONLY	BM25 only	Exact term matching
SPLADE	SPLADE learned sparse only	Semantic term expansion without dense vectors
SPLADE_HYBRID	SPLADE + Dense vector	Best of sparse + dense
COLBERT_RERANK	Hybrid + ColBERT late-interaction reranking	Maximum retrieval quality
FULL_STACK	Vector + BM25 + SPLADE + ColBERT reranking	When recall quality matters more than latency

The Scoring Formula (MF-001 Appendix A)

Each signal contributes to the final score, with floor constraints ensuring no soft signal can eliminate a trace before fusion:

• sim(C,T) -- Semantic similarity (cosine or Euclidean-based)
• In(T) -- Normalized importance (0, 1], scaled from the encoding-time salience score
• D(T) -- Retrieval strength: decays with disuse, rises with recall (current accessibility)
• S(T) -- Storage strength: encoding durability, non-decreasing except on tombstone
• V(T,C) -- Valence alignment within the cue's emotional window
• tag(C,T) -- Tag containment boost via Bloom filter pre-filtering
• G(T,C) -- Graph activation boost from Hebbian co-activation and entity links

Floor constraints prevent elimination: D in [0.10, 1.0], V in [0.01, 1.0], G >= 1.0, S >= 1.0.

---

Score Provenance in Recall Results

Every recall result includes a full score breakdown so you can inspect exactly why a memory ranked where it did:

Score: 0.0325
Score Breakdown:
  similarity:    0.9269
  imp x decay:   1.9425
  tag_boost:     1.00x
  habituation:   1.00x
  graph_boost:   1.00x
  valence_align: 1.00x
Provenance:
  confidence: 0.36
  age_days: 19.0
  importance: 5.33
  memory_type: SEMANTIC
  recall_count: 0
  decay_factor: 0.364

---

Discussion Questions

1. Have you hit the truncation trap in production? What retrieval strategies did you use to work around it?
2. What retrieval modes have you tried in your agent pipelines? Dense-only, hybrid, or something else?
3. What is your experience with ColBERT late-interaction reranking vs standard bi-encoder retrieval? Worth the latency cost?
4. How do you handle scoring transparency in your current setup? Do you expose score breakdowns to end users or agent loops?
5. At what scale (number of memories) does single-signal retrieval start failing consistently in your experience?

[Rehanguards]

Great breakdown on transport modes! When dealing with Streamable HTTP vs REST for agent memory/tools, we often see authentication and schema parsing become the main latency bottlenecks under high concurrency. Have you benchmarked the P99 overhead when adding auth middleware/guardrails over the HTTP transport layer?

Thanks for jumping in — useful angle, even though this thread is mostly about multi-signal recall vs the truncation trap.

On the transport side: Spector’s MCP layer supports stdio for local agents and Streamable HTTP for remote/cloud. Synapse also exposes REST, with optional JWT / API-key auth and per-user memory isolation when spector.auth.enabled is on (see PR #335). Auth sits in front of the request; it doesn’t change the fused scoring path once a recall call is authorized.

We haven’t published a dedicated P99 breakdown of auth middleware vs schema parsing on Streamable HTTP under high concurrency yet. In practice we keep that path deliberately thin (hashed API-key / JWT validation before the memory tools run), and the bigger latency swings we care about tend to show up in embedding + multi-signal candidate generation, not the auth gate.

If you’ve got numbers from your setup (auth on/off, REST vs Streamable HTTP, concurrency level), I’d love to see them — happy to compare notes, and it would help prioritize what we should instrument next.

Curious whether the auth/schema tax you’re seeing is mostly cold-start (session init / schema fetch) or per-tool-call under steady load?

[Rehanguards]

Appreciate the detailed breakdown!

From our benchmarks running sub-millisecond ASGI guardrails on high-concurrency Streamable HTTP pipelines, the latency tax mostly hits per-tool-call during steady load rather than cold-start. Specifically:

• Schema & PII Inspection: Recursive JSON parsing on streamed tool inputs introduces a predictable ~0.4ms to 0.8ms overhead per chunk if handled via non-blocking pure ASGI buffers.
• Stream Safety: Standard middleware often causes stream corruption or memory inflation when buffering chunks for schema verification. Keeping the memory footprint capped via early size-limiting keeps P99 spikes minimal under heavy load.

Here is a quick snapshot from our recent benchmark environment (FastAPI / ASGI Event Loop under 1k concurrent tool-call streams):

Metric	Baseline (No Guardrail)	With Cloakwall ASGI Middleware	Latency Tax
P50 Latency	1.2 ms	1.6 ms	+0.4 ms
P99 Latency	4.1 ms	4.9 ms	+0.8 ms
Memory Buffer Cap	Uncapped	Max 10MB (Early-exit 413)	Stable

We built our standalone ASGI middleware (Cloakwall) specifically to solve stream-corruption and latency tax for FastAPI execution pipelines. Happy to compare notes further or share the benchmark harness setup if you're looking into optimizing that path!

[GioiaZheng]

The truncation-trap framing makes sense to me, but I would keep two contracts separate:

1. candidate generation should protect recall across independent signals;
2. fused scoring should decide ranking only after that union exists.

The part I would test hardest is not whether the final formula looks reasonable, but whether each signal gets a real chance to admit candidates before the first cut. Otherwise the system can still recreate the same failure mode with a different name: for example, dense retrieval dominates the candidate budget, and the graph/temporal/importance signal only rescues items that were already semantically close.

For agent memory I would create a small adversarial eval set with cases like:

• low semantic similarity, high importance: old constraints, safety rules, user preferences
• high semantic similarity, low usefulness: recent topical chatter that should not override a durable rule
• lexical match but wrong entity or tenant
• graph-near but temporally stale
• emotionally salient but task-irrelevant

Then report recall before fusion, not only final answer quality:

protected_memory_recall@budget
signal_contribution_rate
rescued_by_non_dense_signal
false_rescue_rate
latency_p50 / latency_p95 by retrieval mode

For the retrieval modes, I would probably keep dense + BM25 as the default baseline, then enable graph/tag/temporal expansion when the agent is operating inside a long-lived memory store. SPLADE and ColBERT are useful, but I would make them policy-driven rather than always-on: run them for ambiguous, high-risk, or low-confidence retrieval, and measure whether the additional rescued memories justify the latency.

On score transparency: I would expose the full score breakdown to the agent loop and logs, but not necessarily to end users. For users I would rather show the reason category, such as "remembered because it is a durable user constraint" or "linked through project X", while keeping the raw multipliers for debugging. Raw fused scores are easy to over-interpret unless they are calibrated.

The most useful regression test would be simple: insert a durable constraint that is old and lexically distant from the current cue, insert a recent semantically similar distractor, and assert that the durable constraint survives candidate generation before reranking. If that invariant holds across corpus growth, the architecture is doing something meaningfully different from ordinary vector-top-k plus rerank.

[Rehanguards]

@GioiaZheng This is an exceptional breakdown—you hit the exact failure mode of naive top-k vector retrieval.

1. Candidate Generation vs. Fused Scoring: Absolutely agree on keeping these contracts separate. If dense vector search consumes the entire candidate budget upfront, downstream reranking is working on a biased union. We are enforcing explicit per-signal candidate quotas (reserving slots for graph/temporal and durable rule signals) before merging and reranking.

2. Conditional Heavy Retrieval: Spot on regarding policy-driven ColBERT/SPLADE execution. Running late-interaction models only on high-ambiguity queries or low-confidence first-pass retrievals keeps latency within p95 bounds without paying a constant penalty.

3. Adversarial Invariant Test: The regression test you suggested—verifying that an old durable constraint survives candidate generation against a semantically similar recent distractor—is brilliant. We are incorporating this exact invariant test into our evaluation harness.

Really appreciate the clarity on protected_memory_recall@budget and user-facing explanation categories!