Skip to content

docs: document the RL and eval substrate in archive/ and legacy/ - #63

Draft
HarleyCoops wants to merge 1 commit into
mainfrom
claude/repo-assessment-ml-tasks-2qv6ti
Draft

docs: document the RL and eval substrate in archive/ and legacy/#63
HarleyCoops wants to merge 1 commit into
mainfrom
claude/repo-assessment-ml-tasks-2qv6ti

Conversation

@HarleyCoops

Copy link
Copy Markdown
Owner

What changed

AGENTS.md described archive/ and legacy/ in one line — "Retired code. Do not import from it; do not 'fix' it." That undersold what those directories actually hold, and it made the repository look substantially smaller than it is to anyone reading the agent guide first.

archive/codex-pipeline/environments/math_to_manim/ is a Prime Intellect Verifiers RL environment (m2m2_visual_repair) with:

  • scoring.py — a weighted static reward over seven components (format, schema, Python parse, static validation, safety, acceptance terms, layout risk)
  • environment.py and nine training/inference/orchestration TOML configs
  • data/repair_tasks.jsonl — the repair-task dataset
  • evals/prompt_suite.yaml — rubric-scored prompt eval cases across five dimensions

legacy/Math-To-Manim/ retains the prior-generation provider pipelines (KimiK2.5Swarm, Gemini3) used as RL baselines and for cross-generation comparison.

This replaces the blanket "do not touch" with the real layout and the rules that actually matter:

  1. No imports from archive/ or legacy/ into mythos/ or sol/ — the RL and eval code depends on the silos, never the reverse.
  2. No casual refactoring of reward functions or task datasets; changing scoring silently invalidates prior runs, so version the schema instead of editing in place.
  3. An explicit note that those artifact schemas have drifted from what mythos/ and sol/ emit today, flagged as deliberate work rather than a bug to patch in passing.

Also adds a one-line summary of what the project does at the top of the guide, which was previously missing.

Why

Point 3 is the substantive one. evals/prompt_suite.yaml scores request_spec, concept_plan, knowledge_tree, math_enrichment, visual_spec, narrative_spec, and scene_spec. The live silos emit 01_intent.json through 06_scene_spec.json. Five of seven artifact names have no counterpart — the eval substrate currently scores a pipeline that no longer exists. Recording that in the agent guide keeps it from being rediscovered or silently "fixed" by renaming keys.

Scope

Documentation only. No code, no behavior change, no test changes.


Generated by Claude Code

The agent guide described archive/ and legacy/ as retired code to be left
alone. That undersold what they hold: archive/codex-pipeline carries a Prime
Intellect Verifiers RL environment (m2m2_visual_repair) with a weighted static
reward function, training and inference configs, a repair-task dataset, and a
rubric-scored prompt eval suite. legacy/ retains the prior-generation
provider pipelines used as RL baselines.

Replace the blanket "do not touch" with the actual layout and the rules that
matter: no imports from archive/ or legacy/ into the runtime silos, no casual
refactoring of reward functions or task datasets, and an explicit note that
those artifact schemas have drifted from what mythos/ and sol/ emit today.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VQpUQYoSDgvMZkVF2NbGiq

Copy link
Copy Markdown
Owner Author

CI is red here, but not because of this PR. Two separate things, neither caused by the diff (which touches only AGENTS.md, and no test reads that file):

1. The cancelled jobs were an infrastructure fluke. test (3.10) and test (3.12) both report conclusion: cancelled after ~15 minutes, but the job objects come back with runner_id: 0, an empty runner_name, and no steps array — they never got a runner and were cancelled while still queued. I've re-run the failed jobs.

2. The real failure is on the base branch and predates this PR. Run #93 on main at fcad0674 — the exact commit this branch is based on — fails the "Run unit tests" step on both Python versions in about two seconds.

Reproduced locally in a clean venv with pip install -e ".[dev]":

ERROR collecting tests/test_mcp_server.py
mythos/mcp_server.py:28: in <module>
    from mcp.server.fastmcp import FastMCP
E   ModuleNotFoundError: No module named 'mcp.server.fastmcp'
...
E   RuntimeError: The MCP server requires the 'mcp' extra: pip install -e '.[mcp]'
!!!! Interrupted: 1 error during collection !!!!

Root cause is dependency drift, not a missing extra. pyproject.toml declares mcp>=1.2 with no upper bound, so CI now resolves mcp 2.0.0, which removed mcp.server.fastmcpFastMCP is not importable from mcp.server either:

mcp.server contents: ['apps.py', 'auth', 'caching.py', 'connection.py', 'context.py',
 'lowlevel', 'mcpserver', 'runner.py', 'session.py', 'stdio.py', ...]   # no fastmcp

The except ImportError guard at mythos/mcp_server.py:28-34 converts this into a RuntimeError at module import time, so it kills pytest collection — the whole suite fails, not just the MCP test. That matches the timeline: run #87 was green on 2026-07-24, #93 red on 2026-07-28, with no intervening code change to that path.

With tests/test_mcp_server.py excluded, the rest of the suite runs. One unrelated flake surfaced — test_sol_staged_pipeline.py::test_codex_streams_jsonl_events_to_trace_and_sink fails in the full run but passes in isolation, alongside a PytestUnraisableExceptionWarning about a ResponseBody object lacking close — which looks like cross-test pollution rather than a real regression.

Fixing this is out of scope for a docs-only PR and the right fix is a judgment call — pin mcp>=1.2,<2 to restore the status quo, or migrate mythos/mcp_server.py to the 2.x server API. I'll leave that to a separate change rather than folding it in here. Happy to open it if useful.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants