Skip to content

Add Tests-fix CI-revert fallback ladder and scratch-file guidance - #67

Merged
dmccoystephenson merged 2 commits into
mainfrom
fix/tests-fix-fallback-and-scratch-file-note
Jul 27, 2026
Merged

Add Tests-fix CI-revert fallback ladder and scratch-file guidance#67
dmccoystephenson merged 2 commits into
mainfrom
fix/tests-fix-fallback-and-scratch-file-note

Conversation

@dmccoystephenson

@dmccoystephenson dmccoystephenson commented Jul 27, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds a fallback ladder to Phase 4's Tests-fix rubric item for when the local test/build anchor cannot run (the UNVERIFIED case): a CI-based temporary revert (push a revert-only commit, confirm CI goes red naming exactly the new tests, restore and confirm green), a pre-existing test whose assertion changed with the fix, or — failing both — score FAIL and hand to a human. Previously the rubric only defined the local git stash experiment and had no substitute, so it silently became unscoreable on exactly the repos (e.g. example-s-dev-loop, confirmed on PR #1976 and #1980) where the local anchor is unavailable and CI is the only anchor.
  • Cross-references the same fallback from the Phase 8 regression gate, which previously named only the stash-and-run as evidence.
  • Adds a Phase 3 Universal rule: prefer the Write tool over > file redirection and python3 -c "import os; os.remove(path)" over rm for scratch-file work, since some harness sandboxes statically block plain > and rm even for files the same session just created (observed on example-s-dev-loop).

Research grounding

Refines the Tests-fix rubric item, which RESEARCH.md's finding §1 ("Self-critique without an external signal is unreliable") already grounds via PR #54 — this PR closes a gap where that grounding had no defined path once the local anchor is unavailable, otherwise silently degrading Tests-fix back to ungrounded judgment on exactly the repos this finding is about. Added an Implementations entry under finding §1.

The scratch-file guidance (Closes #62) is a harness/sandbox operational observation, not literature-backed research — no RESEARCH.md finding applies; per CLAUDE.md's "grounding work in research" section, stating that explicitly rather than forcing a citation.

Test plan

  • python3 scripts/check_docs.py passes (every {{placeholder}} has a substitution-table row; README's "What it does" stays 1:1 with Steps)
  • No new {{placeholder}} introduced; no fenced code blocks added (only prose/markdown lists), so no escaping concerns
  • Manually traced the new Phase 4 text against the existing UNVERIFIED paragraph it cross-references and the Phase 8 regression-gate wording it complements
  • UNVERIFIED — full /create-dev-loop dry run against a fixture repo was not performed this cycle. This session's sandbox restricts filesystem writes to this repo's own working directory, so no external fixture repo could be created to exercise the actual generation path end-to-end (the "no usable target repo is reachable in the environment" case the template's own Phase 4 edge case anticipates). This PR only changes prose inside existing phases (no placeholders/escapes/step-count changes), which lowers the risk, but a maintainer with access to a fixture (example-l-dev-loop / example-q-dev-loop) should confirm the rendered text reads correctly in a freshly generated skill before merge.

Closes #61
Closes #62


drafted by Claude on behalf of Daniel Stephenson

Phase 4's Tests-fix rubric item required a local stash-and-run
experiment with no defined substitute when the local anchor can't run,
even though the UNVERIFIED handling elsewhere in the template fully
anticipates that case; CI on the fixed tree alone can't reproduce the
stashed-revert half. Adds a three-rung fallback ladder (CI-based
temporary revert, a pre-existing test whose assertion changed with the
fix, or FAIL/hand-to-human) and a matching cross-reference from the
Phase 8 regression gate.

Also adds a Phase 3 Universal rule documenting a harness sandbox quirk
(observed on medieval-factions-dev-loop): plain `>` redirection and
`rm` can be statically blocked even for session-created scratch files,
where the `Write` tool and `python3 -c "import os; os.remove(...)"`
are not.

Closes #61
Closes #62
@dmccoystephenson

dmccoystephenson commented Jul 27, 2026

Copy link
Copy Markdown
Member Author

Self-review rubric:

  • Scope: PASS — every changed line is in create-dev-loop.md (Phase 3 Universal rules, Phase 4 Tests-fix rubric, Phase 8 regression gate) or RESEARCH.md (Implementations entry for this PR), all directly tied to Phase 4 Tests-fix has no defined substitute when the local anchor cannot run #61/RESEARCH.md gap: harness sandbox blocks plain '>' redirection and rm even for session-created scratch files #62. No unrelated formatting/renames.
  • Tests-new: N/A — no new code/method added; this is a prose-only template change (no test suite applies to this repo).
  • Tests-fix (empirical, not judged): PASS — git show origin/main:create-dev-loop.md | grep -c "fallback ladder" returns 0 and the equivalent for "Scratch-file handling" also returns 0 on the pre-fix tree (confirms the described gaps were real and absent, analogous to a failing test), while both return non-zero (3 and 1) on the fixed tree. This is the closest empirical analog available in a docs-only repo to the code stash-and-run experiment.
  • Sibling structure: PASS — the new Universal-rules bullet matches the bold-lead-phrase + explanation structure of its two siblings ("Match sibling structure", "Rename siblings together"); the fallback-ladder numbered list matches the existing numbered-list style used elsewhere in Phase 4 step 1's UNVERIFIED paragraph.
  • Sibling renames: N/A — no identifiers renamed.
  • Docs: PASS — python3 scripts/check_docs.py passes (every {{placeholder}} still has its substitution-table row; README's "What it does" list is unaffected since no Step was added/removed). RESEARCH.md finding Align generated skill template with example-l-dev-loop practices #1 has a new Implementations entry recording this PR.
  • Issue resolution: PASS — Phase 4 Tests-fix has no defined substitute when the local anchor cannot run #61's exact named gap (Tests-fix has no substitute when the local anchor can't run) is closed by the fallback ladder; RESEARCH.md gap: harness sandbox blocks plain '>' redirection and rm even for session-created scratch files #62's exact named gap (no guidance for the sandbox blocking redirection/removal) is closed by the new Universal rule, matching the issue's own suggested wording closely.
  • manual validation (this repo's external-signal anchor): UNVERIFIED — this session's sandbox restricts filesystem writes to this repo's own working directory, so no external fixture repo could be created to run /create-dev-loop end-to-end (the template's own "no usable target repo is reachable" edge case). check_docs.py (the one automated check this repo has) passes, and the change is prose-only (no new placeholders/escapes/step-count changes), which lowers risk, but a maintainer with access to a fixture (example-l-dev-loop / example-q-dev-loop) should confirm the rendered text before merge.

Summary: both issues' named gaps are verifiably closed (empirically confirmed absent-before/present-after), but full end-to-end fixture validation could not be run in this sandbox — flagging UNVERIFIED per the template's own scope-gated escape hatch rather than claiming a clean run that was never performed.


drafted by Claude on behalf of Daniel Stephenson

@dmccoystephenson
dmccoystephenson merged commit 658d64a into main Jul 27, 2026
1 check passed
@dmccoystephenson
dmccoystephenson deleted the fix/tests-fix-fallback-and-scratch-file-note branch July 27, 2026 01:50
dmccoystephenson added a commit that referenced this pull request Jul 27, 2026
…nce (#77)

Step 1 told the generator to diff against /tmp/<slug>-dev-loop.new.md
but never said how that file gets written -- the natural reading was
shell redirection, which is exactly what PR #67 documented as blocked
in some sandboxes, and /tmp is outside the repo directory some
sandboxes also restrict. Now names the Write tool and an in-repo
scratch path, and fails closed (abort, don't overwrite unreviewed) if
the write itself fails.

Also updated Step 1's MODE=update bullet, which still said 'record in
my-claude-skills' after PR #69 renamed Step 7 to an optional personal
catalog.

Closes #73
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
* Add Tests-fix CI-revert fallback ladder and scratch-file guidance

Phase 4's Tests-fix rubric item required a local stash-and-run
experiment with no defined substitute when the local anchor can't run,
even though the UNVERIFIED handling elsewhere in the template fully
anticipates that case; CI on the fixed tree alone can't reproduce the
stashed-revert half. Adds a three-rung fallback ladder (CI-based
temporary revert, a pre-existing test whose assertion changed with the
fix, or FAIL/hand-to-human) and a matching cross-reference from the
Phase 8 regression gate.

Also adds a Phase 3 Universal rule documenting a harness sandbox quirk
(observed on medieval-factions-dev-loop): plain `>` redirection and
`rm` can be statically blocked even for session-created scratch files,
where the `Write` tool and `python3 -c "import os; os.remove(...)"`
are not.

Closes #61
Closes #62

* Record PR #67 Implementations entry in RESEARCH.md finding #1
dmccoystephenson added a commit that referenced this pull request Aug 3, 2026
…nce (#77)

Step 1 told the generator to diff against /tmp/<slug>-dev-loop.new.md
but never said how that file gets written -- the natural reading was
shell redirection, which is exactly what PR #67 documented as blocked
in some sandboxes, and /tmp is outside the repo directory some
sandboxes also restrict. Now names the Write tool and an in-repo
scratch path, and fails closed (abort, don't overwrite unreviewed) if
the write itself fails.

Also updated Step 1's MODE=update bullet, which still said 'record in
a-private-repo-3' after PR #69 renamed Step 7 to an optional personal
catalog.

Closes #73
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant