Repository navigation
PersonaMem benchmark: v1 or v2? Request for published results and reproduction steps #87
Description
I'm trying to use your evaluation framework to run PersonaMem but there is some confusion regarding the version.
PersonaMem version
The benchmark chart in the README labels the rightmost panel "PersonaMem v2," but the evaluation code appears to support only PersonaMem v1:
personamem_converter.pyexpectsquestions_32k.csv+shared_contexts_32k.jsonl(the v1 file format from bowen-upenn/PersonaMem)- The converter parses v1-specific columns (
user_question_or_message,shared_context_id,end_index_in_shared_context, etc.) - PersonaMem v2 (bowen-upenn/PersonaMem-v2) uses a different schema (parquet format,
user_querycolumn,incorrect_answersinstead ofall_options, 5,000 benchmark questions vs 588 in v1-32k) - The README links to the v1 HuggingFace dataset for download instructions
Could you clarify which version was used for the published scores (EverMemOS 53.2%, MemOS 50.7%, Mem0 43.9%, Zep 43.4%, memU 38.7% )? If v2 was used, is there a separate converter or branch we're missing?
Published results
For LoCoMo and LongMemEval, the detailed results table in figs/benchmark_2.png is very helpful. Is there a similar breakdown for PersonaMem (per-category accuracy, number of questions, etc.)?
More broadly, are the raw evaluation outputs (answer texts, search results, eval judgments) available for any of the benchmarked systems? Being able to inspect the actual answers would help the community:
- Reproduce the published numbers independently
- Do error analysis across systems
- Ensure fair comparison when benchmarking new systems
Even just the eval_results.json files for each system/dataset combination would be valuable.
Reproduction
We ran PersonaMem through the framework as documented and want to make sure we're comparing apples to apples. A few specific questions:
- Which LLM backbone was used for answer generation in the PersonaMem evaluation? (The LoCoMo/LongMemEval table shows GPT-4o-mini and GPT-4.1-mini backbones — was PersonaMem run with the same?)
-
- How many questions were evaluated: 588 (PersonaMem v1 32k) or 5,000 (PersonaMem v2 benchmark 32k, 128k, 1M)?
- Were any category filters applied (as in LoCoMo category 5)?
Thanks for making the framework open source, it's been very useful. Just want to make sure we're benchmarking on the same footing.
Activity
dial481 commentedon Feb 23, 2026
Following up with some additional context from cross-referencing other published PersonaMem evaluations.
Cross-paper score comparison
I looked at every paper I could find that reports PersonaMem scores for memory systems. The same systems get very different numbers depending on who runs the evaluation:
Sources: BMAM Table 14, O-Mem Table 2.
The spread is significant, MemOS ranges from 34% to 58.7%, Zep from 43.4% to 57.8%, MemU from 38.7% to 56.8%. This suggests PersonaMem scores are highly sensitive to evaluation setup (LLM backbone, answer prompt, context formatting, judging criteria), which makes reproduction details even more important.
Question count
The BMAM paper independently confirms 589 QA pairs across 20 users, which matches the v1-32k download from HuggingFace. (The original PersonaMem paper says 588, the actual CSV has 589 rows.)
Timeline
We also noticed that
personamem_converter.pywas added on Nov 5, 2025 (commit history), while PersonaMem v2 wasn't published until Dec 7, 2025 (arxiv 2512.06688).I'm trying to run benchmarks on PersonaMem through your framework but I'd like to know what is an actual apples to apples comparison. I'd like to ensure the same dataset version, same question count, same LLM backbone, same answer prompt. Without that, any comparison would be apples to oranges, as the cross-paper table above illustrates.
Would it be possible to share the exact evaluation config (prompt, model, parameters) used for the PersonaMem scores in the README?
bestbzw commentedon Mar 16, 2026
Same question here. I noticed that the results reported in the paper are based on PersonaMem v2, but the evaluation script currently available in this repository is for v1. Looking forward to the release of the v2 evaluation scripts!
Celina-love-sweet commentedon Mar 21, 2026
This work should be using PersonsMem-V1-32K (588 QA). When I evaluated the local version of Mem0 on PersonsMem-V1-32K (588 QA), I obtained an accuracy of 65.5%, which appears to significantly exceed the currently reported results. I will update the results after testing MemOS and EverMemOS.
Celina-love-sweet commentedon Mar 21, 2026
May I ask if you have evaluated Mem0 on PersonaMem-V1-32K? I obtained an accuracy of 65.5%, which seems significantly higher than the results reported in the literature.
dial481 commentedon Mar 21, 2026
@Celina-love-sweet no I evaluated Penfield.. If I recall correctly the score was %67.x. I did not publish the results because it wasn't clear whether v1 or v2 was being used.
ngocbh commentedon May 17, 2026
Has anyone reproduced PersonaMem-V2 reported in the paper? I see some PROFILE_UPDATE_PROMPT in the source code, but not sure if it's used in evaluation.
LivXue commentedon May 24, 2026
Some guy from Evermind told me that a colleague conducted the evaluation on PersonaMem V2 but the code is not merged into this repository. I am also attempting to reproduce the results.
gloryfromca commentedon Oct 13, 2026
PersonaMem is no longer part of this repository's benchmark suite —
personamem_converter.pyand the v1 configs are gone, and the README benchmark chart you screenshotted was replaced. The suite is now four benchmarks (LoCoMo, LongMemEval, EverMemBench, SubtleMemory) behind one runner (#425), each with its published number, its exact config and areproduce.shthat starts and stops its own server: benchmarks/README.md.Your underlying point about reproducibility is taken: every run now writes
run_spec.json(models actually served, endpoints, package versions), per-questionanswer_*.jsonl/judge_*.jsonl, and per-category lines inreport.txt, and the README states plainly where our headline statistic differs from how other papers compute theirs.Closing because the v1/v2 question can no longer be answered from this repository's code. Thanks for the cross-paper comparison table — that framing is why the current suite records its configuration.