Skip to content

README.md

Latest commit

 

History

History
306 lines (228 loc) · 9.1 KB

README.md

File metadata and controls

306 lines (228 loc) · 9.1 KB
Raw

Agent Memory Evaluation

中文版

Documentation index

Agent Memory Evaluation is the AgentBench-based evaluation track in OmniMemEval. It measures the task performance of an agent runtime after a memory plugin is installed. The current implementation evaluates OpenClaw across five task domains and supports both plain agent runs and memory-plugin lifecycle runs.

Results are recorded in results.md. The current architecture and extension contracts are documented in architecture.md.

Evaluation Protocols

  • plain AgentBench: runs test_only or train_then_test without enabling the target memory plugin.
  • memory plugin AgentBench: runs the memory_train_backup_test protocol, including memory cleanup, train execution, memory settling, backup, restore, and test execution.

Data Source

AgentBench uses the EvoAgentBench dataset published by EverMind (Hugging Face/GitHub namespace: EverMind-AI). Dataset ownership, task splits, and upstream benchmark definitions follow the EverMind-AI/EvoAgentBench dataset and project documentation. OmniMemEval only provides runner integration, configuration, lifecycle management, and result organization.

References:

  • Hugging Face dataset: EverMind-AI/EvoAgentBench
  • GitHub project: EverMind-AI/EvoAgentBench
  • Underlying benchmarks: BrowseCompPlus, OmniMath, SWE-Bench, LiveCodeBench, GDPVal

Dataset size:

Domain Benchmark Train Test
information_retrieval BrowseCompPlus 154 65
reasoning OmniMath 478 100
software_engineering SWE-Bench 101 26
code_implementation LiveCodeBench 97 39
knowledge_work GDPVal 87 58

Environment

Agent Memory Evaluation uses a separate conda environment from the User Memory track:

conda create -n agentmem python=3.12 -y
conda activate agentmem
python -m pip install -U pip
pip install -r requirements_agentbench.txt

Do not reuse the User Memory environment for AgentBench. User Memory and AgentBench dependencies are maintained in separate requirements files.

Install OpenClaw CLI and make sure it is available in PATH:

npm install -g openclaw

For memory-plugin lifecycle evaluations, install the target memory plugin before running AgentBench. For example, install the MemOS local plugin with:

curl -fsSL https://raw.githubusercontent.com/MemTensor/MemOS/main/apps/memos-local-plugin/install.sh | bash

After installation, configure OpenClaw and the installed memory plugin before running validation. The exact plugin configuration depends on the plugin under evaluation. For the MemOS local plugin, complete the plugin configuration generated by the installer under the OpenClaw configuration directory, then confirm that the plugin entry is enabled in OpenClaw.

At minimum, verify the following before running AgentBench:

  • OpenClaw has a valid model/provider configuration.
  • The memory plugin is installed and enabled in the OpenClaw config.
  • Required plugin credentials, local paths, or service endpoints are configured.
  • The AgentBench env file .env.agent is prepared as described below.

After installation and configuration are complete, verify that the OpenClaw CLI can be invoked normally:

openclaw --version
openclaw agent --help

Create the Agent Memory env file from the template:

cp env_examples/.env.agent .env.agent

AgentBench loads .env.agent by default. --env FILE can be used to supply an additional env file. This keeps Agent Memory credentials separate from User Memory backend env files such as .env.memos and .env.mem0.

OpenClaw model settings are configured in:

configs/agentbench/agents/openclaw.yaml

Common .env.agent fields:

LLM_BASE_URL=...
LLM_API_KEY=...

JUDGE_MODEL=...
JUDGE_API_BASE=...
JUDGE_API_KEY=...

IR_EMBEDDING_ENDPOINT=...
IR_EMBEDDING_MODEL=...
IR_EMBEDDING_API_KEY=...

EVALUATION_API_BASE=...
EVALUATION_API_KEY=...
EVALUATION_MODEL_OWNER=...
EVALUATION_MODEL_NAME=...
EVALUATION_TIMEOUT=240
EVALUATION_MAX_RETRIES=3

OpenClaw model/provider credentials are injected from .env.agent into configs/agentbench/agents/openclaw.yaml. If provider fields are not set there, the adapter falls back to values resolvable from ~/.openclaw/openclaw.json.

Data Preparation

Run the following commands from the repository root:

mkdir -p data/agentbench
huggingface-cli download EverMind-AI/EvoAgentBench \
  --repo-type dataset \
  --local-dir ./data/agentbench

Expected layout:

data/agentbench/
  BrowseComp-Plus/
  Reasoning & Problem Decomposition/
  gdpval/
  livecode/
  swebench/

Domain data paths are configured under configs/agentbench/domains/*.yaml. If a custom data location is used, update the relevant domain yaml fields.

LiveCodeBench requires its official verifier package:

git clone https://github.com/LiveCodeBench/LiveCodeBench.git ./LiveCodeBench
pip install --no-deps -e ./LiveCodeBench

BrowseComp-Plus dense retrieval requires preprocessing, an index, and embedding service configuration:

python scripts/agentbench/utils/browsecomp-plus-tools/setup_data.py \
  --output-dir ./data/agentbench/BrowseComp-Plus \
  --skip-index
python scripts/agentbench/utils/browsecomp-plus-tools/build_dense_index.py \
  --config configs/agentbench/domains/information_retrieval.yaml

SWE-Bench requires Docker. GDPVal requires PDF/Office processing utilities. On Debian/Ubuntu:

apt-get update
apt-get install -y docker.io poppler-utils libreoffice openjdk-21-jdk
systemctl start docker

Quick Start

Reasoning smoke run:

cp env_examples/.env.agent .env.agent
mkdir -p data/agentbench
huggingface-cli download EverMind-AI/EvoAgentBench \
  --repo-type dataset \
  --local-dir ./data/agentbench
./scripts/run_agent_eval.sh \
  --agent openclaw \
  --domain reasoning \
  --protocol test_only \
  --version smoke_agentbench \
  --trials 1 \
  --parallel 1

Protocols

test_only

Runs the test split.

./scripts/run_agent_eval.sh \
  --agent openclaw \
  --domain reasoning \
  --protocol test_only \
  --version baseline_reasoning_test \
  --trials 1 \
  --parallel 1

train_then_test

Runs train split followed by test split. This protocol does not execute memory plugin lifecycle commands.

./scripts/run_agent_eval.sh \
  --agent openclaw \
  --domain reasoning \
  --protocol train_then_test \
  --version train_then_test_reasoning \
  --trials 1 \
  --parallel 1

memory_train_backup_test

This protocol evaluates memory plugins separately from baseline results:

  1. Set plugin to train mode.
  2. Clear memory.
  3. Run train split.
  4. Submit verifier feedback as the next turn in the same OpenClaw session.
  5. If structured feedback is enabled in plugin config, submit plugin-side explicit feedback while preserving the same session/episode semantics.
  6. Wait for memory settling/evolution.
  7. Back up domain memory.
  8. Restore the domain backup before every test run.
  9. Run test split and write results to results/agentbench.

Single domain:

./scripts/run_agent_eval.sh \
  --agent openclaw \
  --domain reasoning \
  --protocol memory_train_backup_test \
  --memory-plugin memos \
  --version memos_reasoning_eval \
  --test-runs 1 \
  --trials 1 \
  --parallel 1

Five domains:

./scripts/run_agentbench_memory_train_backup_test.sh \
  --memory-plugin memos \
  --version memos_5domain_eval \
  --test-runs 1 \
  --trials 1 \
  --parallel 1

Domain subset:

./scripts/run_agentbench_memory_train_backup_test.sh \
  --memory-plugin memos \
  --domains reasoning,code_implementation \
  --version memos_reasoning_code_eval

Memory Plugin Configuration

Plugin lifecycle configs live under:

configs/agentbench/memory_plugins/
  memos.yaml
  everos.yaml
  openviking.yaml
  supermemory.yaml
  hindsight.yaml

Each config declares:

  • plugin: plugin label used in result directory names.
  • backup_dir and backup_file_template: memory backup location.
  • settle_seconds or commands.wait_settle: post-train memory settling logic.
  • home_links: plugin paths linked into the isolated OpenClaw home.
  • modes.train/test or commands.set_mode_*: plugin read/write mode.
  • commands.clear/backup/restore: memory cleanup, backup, and restore commands.
  • feedback: train feedback settings and optional plugin-side structured feedback.

Results

Each domain writes outputs under:

results/agentbench/<profile>-<version>-<domain>/
  experiment_config.json
  memory_lifecycle.log
  memory_lifecycle.json
  train/
  test_run_1/

Key files:

  • summary.json: pass@1, average reward, and average runtime.
  • result.json.agent_result: OpenClaw completion status and session recovery metadata.
  • result.json.feedback_result: train feedback turn status.
  • result.json.memos_feedback_result: plugin-side structured feedback status when enabled.
  • memory_lifecycle.json: clear/backup/restore lifecycle events.

Recent evaluation results are recorded in results.md. Model service, external judge service, and embedding service status may affect per-sample rewards.