docs: document the RL and eval substrate in archive/ and legacy/ - #63
docs: document the RL and eval substrate in archive/ and legacy/#63HarleyCoops wants to merge 1 commit into
Conversation
The agent guide described archive/ and legacy/ as retired code to be left alone. That undersold what they hold: archive/codex-pipeline carries a Prime Intellect Verifiers RL environment (m2m2_visual_repair) with a weighted static reward function, training and inference configs, a repair-task dataset, and a rubric-scored prompt eval suite. legacy/ retains the prior-generation provider pipelines used as RL baselines. Replace the blanket "do not touch" with the actual layout and the rules that matter: no imports from archive/ or legacy/ into the runtime silos, no casual refactoring of reward functions or task datasets, and an explicit note that those artifact schemas have drifted from what mythos/ and sol/ emit today. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01VQpUQYoSDgvMZkVF2NbGiq
|
CI is red here, but not because of this PR. Two separate things, neither caused by the diff (which touches only 1. The cancelled jobs were an infrastructure fluke. 2. The real failure is on the base branch and predates this PR. Run #93 on Reproduced locally in a clean venv with Root cause is dependency drift, not a missing extra. The With Fixing this is out of scope for a docs-only PR and the right fix is a judgment call — pin Generated by Claude Code |
What changed
AGENTS.mddescribedarchive/andlegacy/in one line — "Retired code. Do not import from it; do not 'fix' it." That undersold what those directories actually hold, and it made the repository look substantially smaller than it is to anyone reading the agent guide first.archive/codex-pipeline/environments/math_to_manim/is a Prime Intellect Verifiers RL environment (m2m2_visual_repair) with:scoring.py— a weighted static reward over seven components (format, schema, Python parse, static validation, safety, acceptance terms, layout risk)environment.pyand nine training/inference/orchestration TOML configsdata/repair_tasks.jsonl— the repair-task datasetevals/prompt_suite.yaml— rubric-scored prompt eval cases across five dimensionslegacy/Math-To-Manim/retains the prior-generation provider pipelines (KimiK2.5Swarm, Gemini3) used as RL baselines and for cross-generation comparison.This replaces the blanket "do not touch" with the real layout and the rules that actually matter:
archive/orlegacy/intomythos/orsol/— the RL and eval code depends on the silos, never the reverse.mythos/andsol/emit today, flagged as deliberate work rather than a bug to patch in passing.Also adds a one-line summary of what the project does at the top of the guide, which was previously missing.
Why
Point 3 is the substantive one.
evals/prompt_suite.yamlscoresrequest_spec,concept_plan,knowledge_tree,math_enrichment,visual_spec,narrative_spec, andscene_spec. The live silos emit01_intent.jsonthrough06_scene_spec.json. Five of seven artifact names have no counterpart — the eval substrate currently scores a pipeline that no longer exists. Recording that in the agent guide keeps it from being rediscovered or silently "fixed" by renaming keys.Scope
Documentation only. No code, no behavior change, no test changes.
Generated by Claude Code