Question about ALFWorld adaptation for baselines

[FredJiang0324]

Hi, thanks for releasing this impressive work.

I have a question about the Embodied Interactive Tasks(ALFWorld) experiments for baselines such as A-MEM and MemoryOS. Since these methods were originally designed more for dialogue / QA settings, I was wondering how you adapted them to ALFWorld in practice.
In particular:
How did you build their memory bank from the ALFWorld training trajectories? What did you use as the retrieval query during evaluation (objective only, or also observation/history)?
Was the adaptation mainly prompt-level, or did you also modify some code structure / memory pipeline for these baselines?

I’m trying to understand whether the ALFWorld setup for these baselines was mostly a unified prompting interface, or whether some method-specific implementation changes were needed.

Thanks again, the work is very impressive.

[FredJiang0324]

Also do you plan to release any additional code for how baselines were tested on ALFWorld and the evolved MemSkill's skills?

[ViktorAxelsen]

Thank you for your interest in our work and for the thoughtful questions.

You are absolutely right that methods such as A-MEM and MemoryOS were originally designed more for dialogue and QA-style settings. Since we also evaluate on LoCoMo and LongMemEval, we included these baselines in ALFWorld as part of a unified comparison setup for presentation consistency.

For the adaptation, we followed the original methods as closely as possible. We did not modify their core code structure or redesign their memory pipeline. Instead, we only made the minimal adaptations necessary to ensure consistent experimental settings across benchmarks. In this sense, the adaptation was mainly at the prompt level rather than through major method-specific implementation changes.

For the ALFWorld evaluation pipeline, it is almost the same as src/alfworld_env_runner.py in this repo, except that the injected memory is constructed by each baseline’s own memory summarization mechanism. In other words, the surrounding agent-environment interaction pipeline remains largely the same, while the main difference lies in how the memory content is produced and then provided back to the agent.

As for the evolved MemSkill skills, they have already been released both in the paper appendix and in the skills directory of this repository.

Regarding additional code release, we do plan to release more ALFWorld results with other baselines such as AWM and Expel as well. We agree that this would make the comparison more complete and helpful for the community.

Thanks again for the careful reading and for raising these questions.

[Euclider]

Thanks for the clarification. I think I understand the ALFWorld setup better now, but I want to confirm one concern.

From the paper and src/alfworld_env_runner.py, my understanding is that the agent-environment interaction pipeline is fixed, and the main difference across methods is only how the injected memory is constructed. In particular, the memory used in each ALFWorld episode seems to be prepared before rollout, rather than dynamically updated as the current trajectory unfolds.

If that is correct, then there may be a risk that the injected memory contains instance-specific factual knowledge from other trajectories, such as object locations or states, instead of only reusable procedural guidance. In embodied tasks like ALFWorld, such factual memory may not transfer well to a new episode and could even mislead the agent.

So I wanted to confirm:

1. During ALFWorld evaluation, is the injected memory fixed before the episode starts, rather than updated online step by step?
2. If yes, how do you ensure that the injected memory remains general procedural guidance instead of trajectory-specific factual memory?
3. Should the main transferable part be the learned skills / summarization behavior, rather than the memory content itself?

Thanks again for the helpful explanation.