Should Recall Support Optional Query Rewrite?

[Sanderhoff-alt]

Summary

Hindsight's recall currently sends the user's query directly into the retrieval pipeline. The query is sanitized, embedded, and used across hybrid retrieval strategies such as semantic search, BM25, graph traversal, temporal retrieval, fusion, reranking, MMR, and token filtering.

This keeps recall fast and predictable. However, some recall failures may come from query formulation rather than retrieval itself. It may be worth discussing whether recall should support an optional query rewrite or retrieval planning mechanism for ambiguous, conversational, underspecified, or multi-hop memory questions.

Current Behavior

The current recall flow is roughly:

user query
  -> sanitize query
  -> generate query embedding
  -> semantic + BM25 + graph + temporal retrieval
  -> fusion / reranking / MMR
  -> token-budget filtering

In reflect, the agent can already generate tool parameters such as:

search_mental_models(query=...)
search_observations(query=...)
recall(query=...)

So reflect has some implicit agentic query formulation. Plain recall, however, does not have a dedicated, traceable, configurable query rewrite step.

Why This Might Be Useful

Optional query rewrite could help when:

• The user asks a conversational question with missing context, such as "what did we decide last time?"
• The query contains vague references, such as "that migration issue" or "the customer from yesterday".
• A broad question would benefit from multiple targeted retrieval queries.
• Recall needs entities, dates, project names, or fact-type hints that are implicit in the user's wording.
• A high-budget recall or mental model refresh is willing to trade extra cost for better retrieval quality.

At the same time, query rewrite can be risky for memory systems. If a rewrite invents context, over-specifies the query, or chooses the wrong entity/time range, retrieval may silently move into the wrong topic.

Possible Direction

A conservative direction would be to make query rewrite optional and additive:

• Keep the original query.
• Optionally generate one or more retrieval-friendly query candidates.
• Make the generated queries visible in traces.
• Fall back to the original query if rewriting fails or is disabled.
• Keep low-latency recall unchanged by default unless maintainers prefer otherwise.

Rather than replacing the query with one rewritten string, Hindsight might eventually consider a small retrieval plan that preserves the original query and records why any extra candidates were generated.

Open Questions

• Should this belong in recall, reflect, or both?
• Should it be controlled by request parameter, bank config, environment variable, or budget level?
• Should rewrite be disabled by default and only enabled for mid / high budget calls?
• Should Hindsight generate a single rewritten query or multiple retrieval candidates?
• How should generated queries be attributed in recall traces?
• How should the project evaluate whether this improves recall quality enough to justify the added cost and latency?

Expected Outcome of This Discussion

The goal is to decide whether optional query rewrite is worth exploring for recall, and if so, what constraints would make it safe enough for long-term memory retrieval.

[nicoloboschi]

the design of Hindsight is explictly:

• recall = search, fast and predictable
• reflect = anything smart

as you mentioned, reflect already performs query rewriting.

also, since Hindsight is usually queried by LLM/Agents, they get the right query format in the first place.

you can always implement query rewriting at application/client level