Add Tests-fix CI-revert fallback ladder and scratch-file guidance - #67
Merged
dmccoystephenson merged 2 commits intoJul 27, 2026
Merged
Conversation
Phase 4's Tests-fix rubric item required a local stash-and-run experiment with no defined substitute when the local anchor can't run, even though the UNVERIFIED handling elsewhere in the template fully anticipates that case; CI on the fixed tree alone can't reproduce the stashed-revert half. Adds a three-rung fallback ladder (CI-based temporary revert, a pre-existing test whose assertion changed with the fix, or FAIL/hand-to-human) and a matching cross-reference from the Phase 8 regression gate. Also adds a Phase 3 Universal rule documenting a harness sandbox quirk (observed on medieval-factions-dev-loop): plain `>` redirection and `rm` can be statically blocked even for session-created scratch files, where the `Write` tool and `python3 -c "import os; os.remove(...)"` are not. Closes #61 Closes #62
Member
Author
|
Self-review rubric:
Summary: both issues' named gaps are verifiably closed (empirically confirmed absent-before/present-after), but full end-to-end fixture validation could not be run in this sandbox — flagging UNVERIFIED per the template's own scope-gated escape hatch rather than claiming a clean run that was never performed. drafted by Claude on behalf of Daniel Stephenson |
dmccoystephenson
deleted the
fix/tests-fix-fallback-and-scratch-file-note
branch
July 27, 2026 01:50
This was referenced Jul 27, 2026
dmccoystephenson
added a commit
that referenced
this pull request
Jul 27, 2026
…nce (#77) Step 1 told the generator to diff against /tmp/<slug>-dev-loop.new.md but never said how that file gets written -- the natural reading was shell redirection, which is exactly what PR #67 documented as blocked in some sandboxes, and /tmp is outside the repo directory some sandboxes also restrict. Now names the Write tool and an in-repo scratch path, and fails closed (abort, don't overwrite unreviewed) if the write itself fails. Also updated Step 1's MODE=update bullet, which still said 'record in my-claude-skills' after PR #69 renamed Step 7 to an optional personal catalog. Closes #73
dmccoystephenson
added a commit
that referenced
this pull request
Aug 3, 2026
* Add Tests-fix CI-revert fallback ladder and scratch-file guidance Phase 4's Tests-fix rubric item required a local stash-and-run experiment with no defined substitute when the local anchor can't run, even though the UNVERIFIED handling elsewhere in the template fully anticipates that case; CI on the fixed tree alone can't reproduce the stashed-revert half. Adds a three-rung fallback ladder (CI-based temporary revert, a pre-existing test whose assertion changed with the fix, or FAIL/hand-to-human) and a matching cross-reference from the Phase 8 regression gate. Also adds a Phase 3 Universal rule documenting a harness sandbox quirk (observed on medieval-factions-dev-loop): plain `>` redirection and `rm` can be statically blocked even for session-created scratch files, where the `Write` tool and `python3 -c "import os; os.remove(...)"` are not. Closes #61 Closes #62 * Record PR #67 Implementations entry in RESEARCH.md finding #1
dmccoystephenson
added a commit
that referenced
this pull request
Aug 3, 2026
…nce (#77) Step 1 told the generator to diff against /tmp/<slug>-dev-loop.new.md but never said how that file gets written -- the natural reading was shell redirection, which is exactly what PR #67 documented as blocked in some sandboxes, and /tmp is outside the repo directory some sandboxes also restrict. Now names the Write tool and an in-repo scratch path, and fails closed (abort, don't overwrite unreviewed) if the write itself fails. Also updated Step 1's MODE=update bullet, which still said 'record in a-private-repo-3' after PR #69 renamed Step 7 to an optional personal catalog. Closes #73
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
git stashexperiment and had no substitute, so it silently became unscoreable on exactly the repos (e.g.example-s-dev-loop, confirmed on PR #1976 and #1980) where the local anchor is unavailable and CI is the only anchor.Writetool over> fileredirection andpython3 -c "import os; os.remove(path)"overrmfor scratch-file work, since some harness sandboxes statically block plain>andrmeven for files the same session just created (observed onexample-s-dev-loop).Research grounding
Refines the Tests-fix rubric item, which RESEARCH.md's finding §1 ("Self-critique without an external signal is unreliable") already grounds via PR #54 — this PR closes a gap where that grounding had no defined path once the local anchor is unavailable, otherwise silently degrading Tests-fix back to ungrounded judgment on exactly the repos this finding is about. Added an Implementations entry under finding §1.
The scratch-file guidance (Closes #62) is a harness/sandbox operational observation, not literature-backed research — no RESEARCH.md finding applies; per CLAUDE.md's "grounding work in research" section, stating that explicitly rather than forcing a citation.
Test plan
python3 scripts/check_docs.pypasses (every{{placeholder}}has a substitution-table row; README's "What it does" stays 1:1 with Steps){{placeholder}}introduced; no fenced code blocks added (only prose/markdown lists), so no escaping concerns/create-dev-loopdry run against a fixture repo was not performed this cycle. This session's sandbox restricts filesystem writes to this repo's own working directory, so no external fixture repo could be created to exercise the actual generation path end-to-end (the "no usable target repo is reachable in the environment" case the template's own Phase 4 edge case anticipates). This PR only changes prose inside existing phases (no placeholders/escapes/step-count changes), which lowers the risk, but a maintainer with access to a fixture (example-l-dev-loop/example-q-dev-loop) should confirm the rendered text reads correctly in a freshly generated skill before merge.Closes #61
Closes #62
drafted by Claude on behalf of Daniel Stephenson