Extraction quality and chunking improvements

[noctuid]

My experience is that without paying attention to chunking and the context field, I get poor extraction quality/accuracy. This helps:

• context field - am now setting with far more detailed bullet points for any global context that is necessary for clarification
• always passing json so that chunking does not split messages and so date, role, tool name/file path, etc. info is preserved - helps tremendously

Needing to set more global context takes effort though (even though the agent can handle it to an extent), and there are still cases this doesn't handle, including anything where more local context is needed. For example, I'll often reply to numbered agent questions or review points with numbers myself, and while the number is enough context for the agent in-session, there's not necessarily enough info for extraction (previous messages not available).

It seems to me that the main benefit of the current chunking is that it can be easily parallelized, though I would sacrifice that for higher quality extraction.

Obviously the smaller models used for extraction may not be able to as effectively function at higher levels of context, so I think smaller more focused chunking does make sense. For some other memory system, I might fork off an agent with the same model as the main session every turn to extract new memories. I don't think that really makes sense for hindsight, but I do wonder if it might make sense to optionally provide some sort of extra context from previous parts of the document. This wouldn't have be the full prior document, it could be something less:

• the actual previous n chunk(s) (as context and model told not to extract from them) - don't know how well this would work; could potentially benefit from prompt caching (e.g. feed one chunk then the next and so on up to some max context with a stable prefix)
• maybe some previously extracted memories from the document
• maybe just smaller "working context" information returned from previous chunks - in addition to being asked to extract memories, the model would extract things, e.g.
  • "currently discussing blog post" and that description would be updated if the topic transitioned
  • active numbered options
  • open state
• some sort of ambiguity flagging - could still allow for parallelism
  • more context could be requested/retrieved if needed for any given chunk
  • or maybe alternatively some sort of second pass only for flagged chunks; maybe the new curation is relevant here, though I am thinking about something fully automatic

I realize this is not so simple to address, but I am curious what alternative anyone has considered, or if any other memory systems are doing things differently.

[noctuid]

I'm going to add notes about how other memory systems do extraction currently as ideas.

@nicoloboschi This might be interesting to you. Hydradb claims to beat hindsight on BEAM (though only BEAM 1M results are reported). That paper seems very relevant here. They use a sliding window. This is kind of like what I mentioned (including previous chunk(s) for context but not extraction), except they also include segments after the main segment.