Skip to content

Exclude SkillOpt-generated Codex replay sessions from harvest - #287

Open
kaluli123123 wants to merge 1 commit into
microsoft:mainfrom
kaluli123123:fix/codex-harvest-self-replay-286
Open

kaluli123123 wants to merge 1 commit into
microsoft:mainfrom
kaluli123123:fix/codex-harvest-self-replay-286

Conversation

@kaluli123123

Copy link
Copy Markdown

Summary

  • Exclude SkillOpt-generated Codex replay sessions from harvest_codex.
  • Reuse the existing agent-session filter and add narrow Codex replay markers.
  • Preserve ordinary user Codex sessions with regression coverage.

Root cause

CodexCliBackend.attempt_with_tools() emits internal prompts containing SkillOpt skill, memory, task, judge, and reflection instructions. harvest_codex() did not identify those sessions, so later harvesting could treat SkillOpt's own headless work as user evidence.

Changed behavior

Codex harvesting now skips sessions marked as agent sessions or matching the explicit SkillOpt replay prompt shape. It does not apply the generic short-duration headless heuristic, preserving valid short user sessions.

Test plan

  • python3 -m unittest -v tests.test_harvest_codex_replay tests.test_sleep_engine.TestHarvest — 19 passed.
  • python3 -m compileall -q skillopt_sleep tests/test_harvest_codex_replay.py — passed.
  • git diff --check — passed.
  • python3 -m pytest -q — not run successfully because pytest is not installed/available in the environment.

Issue

Fixes #286

Incomplete gates

Full pytest, CI, maintainer review, and merge remain pending. No merge or Issue closure was performed.

@Yif-Yang

Copy link
Copy Markdown
Contributor

Thank you for tackling the self-harvest feedback loop; preventing generated sessions from becoming user evidence is important.

Reviewed 4fabeeea0821.

skillopt_sleep/harvest_codex.py:35: The current Codex backend's ordinary attempt, judge, and reflect prompts are not recognized here. Capturing the actual calls and feeding their prompts through a synthetic archived session leaves all three in harvest: they start Complete the following task for the user., Score how well the response satisfies the rubric, and You are SkillOpt's optimizer.. Only attempt_with_tools is excluded. Please cover the actual emitted prompts (preferably with an explicit engine-origin marker) and add backend-capture regressions before closing #286.

skillopt_sleep/harvest_codex.py:45: These substring checks also discard ordinary user sessions. A first prompt of Please explain the ## CURRENT SKILL section, followed by an assistant response and perfect, thanks, survives main but produces no harvested session on this head; the same happens with an ordinary ## TASK heading. Please use unambiguous engine provenance or a complete generated prompt shape and add preservation coverage for quoted headings/multi-turn sessions.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

skillopt-sleep Codex harvest ingests its own headless replay sessions

2 participants