Skip to content

Commit 27d6cd9

Browse files
committed
feat(memory): explicit opt-in, redaction, namespacing, and a retrieval path
Addresses the review on #118. The integration was previously write-only and enabled by the mere presence of MEM0_API_KEY; both are fixed here. 1. Explicit opt-in. Uploading now requires `mem0_enabled: true`. A key present in the environment for another application no longer enables anything — a disabled run does not even read the key. New `skillopt/memory/settings.py` resolves all `mem0_*` config in one place. 2. Redaction before any outbound request. New `skillopt/memory/redaction.py` is the single choke point: no payload reaches the client without passing through `redact_for_upload`, which strips vendor keys, bearer/basic tokens, JWTs, private keys, `key = value` secret assignments, the project root, and `/home/<user>`-style prefixes. Credentials are scrubbed before paths are collapsed, so a key embedded in a home path cannot survive. Relative paths and filenames are preserved so stored memories remain useful. The patterns deliberately mirror `skillopt_sleep/staging.py` rather than importing it — pyproject keeps that package decoupled with zero research dependency. 3. Stable project-specific namespace. Memories are scoped to `skillopt:<env>:<sha256-prefix-of-project-root>`, which is stable across runs of one project and distinct across projects. The raw path is hashed, never transmitted. Replaces the previous config-name/env user id that could mix unrelated projects. 4. One bounded retrieval call before reflection. `hook_pre_reflect` fetches relevant history and appends it to the reflection input, turning the integration from external logging into a memory loop. It is deliberately assigned to a separate `reflect_context` variable so the autonomous learning-rate decision and the rewrite prompt continue to see the unaugmented context. Also in this commit: - Every call is wall-clock bounded by `mem0_timeout_seconds` (default 5s), so an unreachable service costs one bounded pause instead of blocking a step. - Malformed patches (None, strings, non-lists) are filtered rather than raised inside a swallowing try. - The `mem0` optional extra is declared, deliberately outside `all`. - The `mem0_*` keys are mapped through `_FLATTEN_MAP` under `train.`. Without this they were silently dropped for structured configs — which is every config in the repo, since they all inherit `_base_/default.yaml` — leaving the feature inert with no error. - `configs/features/mem0_memory.yaml` is a documented opt-in example following the `soft_gate.yaml` convention. - The config reference documents every key and exactly what leaves the machine. Tests (`tests/test_mem0_memory.py`, 15 cases, no network) cover: a bare MEM0_API_KEY not enabling uploads, explicit opt-in writing, redaction of secrets and home paths, cap applied after redaction, namespace stability and project separation, structured-config keys surviving flattening, the shipped example config actually enabling the feature, retrieval reaching the actual reflection prompt (verified end-to-end by capturing the optimizer's user message), retrieval being independently disableable, graceful degradation on service failure, timeout bounding, and malformed-patch filtering. Not included, per review guidance: batching, async writes, ranking, and benchmark study are left as follow-up work.
1 parent 905c994 commit 27d6cd9

11 files changed

Lines changed: 1106 additions & 252 deletions

File tree

‎configs/features/mem0_memory.yaml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# ─────────────────────────────────────────────────────────────────────────────
2+
# Feature: optional mem0-backed persistent memory (community-contributed)
3+
# ─────────────────────────────────────────────────────────────────────────────
4+
#
5+
# This is NOT a default SkillOpt setting and was NOT used to produce the
6+
# numbers reported in the paper. It is off unless you turn it on here.
7+
#
8+
# IMPORTANT — this feature sends data to a third-party hosted service.
9+
# Enabling it means skill text and reflection summaries leave your machine.
10+
# Setting MEM0_API_KEY alone does *nothing*; `mem0_enabled: true` below is the
11+
# only thing that turns uploading on, and a key set for some other application
12+
# will never enable it by accident.
13+
#
14+
# What it does:
15+
# - After each evaluation gate, stores the current skill and its score.
16+
# - After each Reflect stage, stores a summary of the patches produced.
17+
# - Before each Reflect stage, makes ONE bounded retrieval call and appends
18+
# the relevant history to that stage's reflection input, so previous runs
19+
# can inform the patches produced now.
20+
#
21+
# When to consider this:
22+
# - You run the same project repeatedly and want later runs to benefit from
23+
# what earlier ones discovered.
24+
# - You are comfortable with the data-handling note above.
25+
#
26+
# When NOT to use this:
27+
# - When reproducing the paper.
28+
# - When the skill text or trajectories are sensitive. Payloads are redacted
29+
# (see docs/reference/config.md → "What leaves the machine"), but redaction
30+
# is a mitigation, not a guarantee about content you know to be private.
31+
#
32+
# Install the optional dependency first:
33+
# pip install 'skillopt[mem0]'
34+
#
35+
# Then export your key and use this file:
36+
# export MEM0_API_KEY=m0-...
37+
# _base_: ../features/mem0_memory.yaml
38+
# or copy the `train:` block below into your config.
39+
# ─────────────────────────────────────────────────────────────────────────────
40+
41+
_base_: ../_base_/default.yaml
42+
43+
train:
44+
# The master switch. Nothing is uploaded while this is false.
45+
mem0_enabled: true
46+
47+
# Optional: set the key in config instead of the MEM0_API_KEY env var.
48+
# Leave empty to use the environment (recommended — keeps keys out of YAML).
49+
mem0_api_key: ""
50+
51+
# Optional: override the namespace. Default is derived per project as
52+
# `skillopt:<env>:<sha256-prefix-of-project-root>`, which is stable across
53+
# runs and distinct across projects. Set this only if you deliberately want
54+
# several checkouts to share one memory pool.
55+
mem0_namespace: ""
56+
57+
# Read memory back into reflection. Setting this false keeps the writes but
58+
# removes the retrieval call — useful for measuring whether retrieval is what
59+
# actually helps, rather than assuming it does.
60+
mem0_retrieval_enabled: true
61+
62+
# Records fetched per retrieval.
63+
mem0_retrieval_limit: 5
64+
65+
# Hard per-call wall-clock bound. On timeout the step continues unchanged, so
66+
# an unreachable service costs one bounded pause rather than a stalled run.
67+
mem0_timeout_seconds: 5.0
68+
69+
# Cap on any single stored payload, applied AFTER redaction so the limit is
70+
# measured on exactly the text that would be transmitted.
71+
mem0_max_chars: 4000

‎docs/reference/config.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,54 @@ defaults to `claude` and can be overridden with `CLAUDE_CLI_BIN`.
138138

139139
Benchmark-specific `env` keys are passed through to the adapter.
140140

141+
## Persistent Memory (mem0) — optional, off by default
142+
143+
Stores skill iterations and reflection summaries in [mem0](https://mem0.ai) and
144+
reads a small amount of relevant history back into the Reflect stage.
145+
146+
**Disabled unless `mem0_enabled` is explicitly true.** A `MEM0_API_KEY` present
147+
in the environment for another application does *not* enable it. Install with
148+
`pip install 'skillopt[mem0]'`.
149+
150+
Set these under `train:` in a structured config (every shipped config is
151+
structured), at the top level of a flat config, or via
152+
`--cfg-options mem0_enabled=true`. A ready-made example ships at
153+
`configs/features/mem0_memory.yaml`.
154+
155+
| Parameter | Type | Default | Description |
156+
|---|---|---|---|
157+
| `train.mem0_enabled` | bool | `false` | Master switch. Nothing is sent unless this is true |
158+
| `train.mem0_api_key` | str | empty | Falls back to `MEM0_API_KEY`, read only when enabled |
159+
| `train.mem0_namespace` | str | derived | Override the namespace; default is derived per project |
160+
| `train.mem0_retrieval_enabled` | bool | `true` | Read memory back into reflection; writes continue if false |
161+
| `train.mem0_retrieval_limit` | int | `5` | Max records fetched per retrieval |
162+
| `train.mem0_timeout_seconds` | float | `5.0` | Hard per-call bound; on timeout training continues |
163+
| `train.mem0_max_chars` | int | `4000` | Cap on any single stored payload, applied after redaction |
164+
165+
### What leaves the machine
166+
167+
When enabled, exactly two record types are sent, both redacted first:
168+
169+
1. **Skill iteration** — epoch, step, score, skill hash and length, the `env`
170+
and model names, and the skill text (capped at `mem0_max_chars`).
171+
2. **Reflection summary** — epoch, step, patch count, rollout scores, and a
172+
JSON summary of up to 10 patches with any `skill_text` field removed.
173+
174+
Retrieval sends the first 600 characters of the current skill as a similarity
175+
query.
176+
177+
Before transmission every payload passes through
178+
`skillopt.memory.redaction.redact_for_upload`, which strips vendor API keys,
179+
bearer/basic tokens, JWTs, private keys, `key = value` secret assignments, the
180+
project root, and `/home/<user>`-style prefixes. Relative paths and filenames
181+
are deliberately preserved so stored memories stay useful.
182+
183+
### Namespacing
184+
185+
Memories are scoped to `skillopt:<env>:<digest>`, where the digest is a SHA-256
186+
prefix of the absolute project root. Stable across runs of one project, distinct
187+
across projects, and the raw path is never transmitted.
188+
141189
## Credential Environment Variables
142190

143191
### Azure-family backend

‎pyproject.toml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,10 @@ searchqa = ["datasets>=2.18.0"]
4646
docs = ["mkdocs-material>=9.5.0", "mkdocstrings[python]>=0.24.0"]
4747
# WebUI dashboard
4848
webui = ["gradio>=4.0.0"]
49+
# Optional mem0-backed persistent memory. Deliberately excluded from `all`:
50+
# installing it is harmless (uploads still require mem0_enabled: true), but
51+
# a data-export integration should be an explicit choice at install time too.
52+
mem0 = ["mem0ai>=0.1.0"]
4953
# Development tools
5054
dev = ["ruff>=0.4.0", "pytest>=8.0.0"]
5155
# All optional dependencies (except docs/dev/webui)

‎skillopt/config.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,14 @@
105105
"train.batch_size": "batch_size",
106106
"train.accumulation": "accumulation",
107107
"train.seed": "seed",
108+
# Optional mem0 memory (off unless train.mem0_enabled is true).
109+
"train.mem0_enabled": "mem0_enabled",
110+
"train.mem0_api_key": "mem0_api_key",
111+
"train.mem0_namespace": "mem0_namespace",
112+
"train.mem0_retrieval_enabled": "mem0_retrieval_enabled",
113+
"train.mem0_retrieval_limit": "mem0_retrieval_limit",
114+
"train.mem0_timeout_seconds": "mem0_timeout_seconds",
115+
"train.mem0_max_chars": "mem0_max_chars",
108116
"gradient.minibatch_size": "minibatch_size",
109117
"gradient.merge_batch_size": "merge_batch_size",
110118
"gradient.analyst_workers": "analyst_workers",

‎skillopt/engine/trainer.py‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,12 @@
7575
set_optimizer_deployment,
7676
)
7777
from skillopt.utils import compute_score, skill_hash
78-
from skillopt.memory.trainer_hooks import maybe_init_mem0, hook_post_evaluate, hook_post_reflect
78+
from skillopt.memory.trainer_hooks import (
79+
maybe_init_mem0,
80+
hook_pre_reflect,
81+
hook_post_evaluate,
82+
hook_post_reflect,
83+
)
7984

8085

8186
# ── Skill-aware reflection: appendix flush ───────────────────────────────────
@@ -1161,11 +1166,18 @@ def _persist_runtime_state(last_completed_step: int) -> None:
11611166
# Build step context from buffer
11621167
step_buffer_context = _format_step_buffer(step_buffer)
11631168

1169+
# Memory read: relevant history for THIS reflection only.
1170+
# Kept in a separate name so the LR decision and rewrite
1171+
# prompts below continue to see the unaugmented context.
1172+
reflect_context = hook_pre_reflect(
1173+
memory, current_skill, step_buffer_context,
1174+
)
1175+
11641176
raw_patches = adapter.reflect(
11651177
rollout_results, current_skill, batch_dir,
11661178
prediction_dir=pred_dir, patches_dir=patches_dir,
11671179
random_seed=batch_seed,
1168-
step_buffer_context=step_buffer_context,
1180+
step_buffer_context=reflect_context,
11691181
meta_skill_context=active_meta_skill,
11701182
)
11711183
failure_patches, success_patches = _normalise_patches(

‎skillopt/memory/__init__.py‎

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,17 @@
1-
"""skillopt.memory — mem0-backed persistent memory for SkillOpt."""
2-
from skillopt.memory.mem0_backend import SkillMemory
1+
"""skillopt.memory — optional, opt-in mem0-backed persistent memory.
32
4-
__all__ = ["SkillMemory"]
3+
Disabled unless a config sets ``mem0_enabled: true``; see
4+
:mod:`skillopt.memory.settings` for the resolution rules and
5+
:mod:`skillopt.memory.redaction` for what is stripped before anything is sent.
6+
"""
7+
from skillopt.memory.mem0_backend import SkillMemory, mem0_available
8+
from skillopt.memory.redaction import redact_for_upload
9+
from skillopt.memory.settings import Mem0Settings, resolve_settings
10+
11+
__all__ = [
12+
"Mem0Settings",
13+
"SkillMemory",
14+
"mem0_available",
15+
"redact_for_upload",
16+
"resolve_settings",
17+
]

0 commit comments

Comments
 (0)