Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 7 additions & 4 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,11 @@ touch the template (e.g. docs-only).
## Test plan

<!--
There is no automated test suite for this repo. Describe how you validated
the change — see CLAUDE.md's "Testing changes" section (running
/create-dev-loop against a real repo and checking the generated skill
compiles, MODE=update behaves correctly, etc.).
CI (.github/workflows/ci.yml) runs scripts/check_docs.py on every PR and
mechanically checks the first two Doc sync boxes above, plus relative links
between the repo's own docs. It catches doc drift, not behavior. There is no
automated behavioral test suite, so describe how you validated the change —
see CLAUDE.md's "Testing changes" section (running /create-dev-loop against
a real repo and checking the generated skill compiles, MODE=update behaves
correctly, etc.).
-->
6 changes: 3 additions & 3 deletions RESEARCH.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ This document records the empirical findings that inform the design of `create-d
- **First-party sources** (Anthropic, OpenAI, vendor research with numbers) count as evidence but must be flagged as first-party so readers can weight them appropriately.
- **Confidence levels**: `high` = replicated across multiple independent studies; `medium` = one well-cited study, plausible; `low` = single paper, contested, or inferred from adjacent literature.

Last reviewed: 2026-07-25.
Last reviewed: 2026-07-27.

---

Expand All @@ -43,7 +43,7 @@ Last reviewed: 2026-07-25.

**Implementations.**
- PR #15 (Reframe self-review as rubric-based and CI-grounded): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. First applied to PR #26 (one cycle, all rubric items PASS); too early to tell whether the rubric catches real defects that free-form review would miss.
- PR #28 (Generalize the Phase 4 external-signal step to support no-CI repos): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Replaces the hardcoded "wait for CI green" anchor with `{{EXTERNAL_SIGNAL_LABEL}}` / `{{EXTERNAL_SIGNAL_CMD}}`, so repos with no CI (including `create-dev-loop` itself) still anchor the rubric on *some* external signal instead of falling back to unanchored judgment — what this finding requires is that an anchor exists, not that it is specifically CI.
- PR #28 (Generalize the Phase 4 external-signal step to support no-CI repos): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Replaces the hardcoded "wait for CI green" anchor with `{{EXTERNAL_SIGNAL_LABEL}}` / `{{EXTERNAL_SIGNAL_CMD}}`, so a repo without CI still anchors the rubric on *some* external signal instead of falling back to unanchored judgment — what this finding requires is that an anchor exists, not that it is specifically CI. `create-dev-loop` itself is the case in miniature: it gained CI in PR #65 (2026-07-27), but that workflow only runs `scripts/check_docs.py`, so the anchor for a *behavioral* change here is still the manual `/create-dev-loop` validation checklist rather than the green check.
- PR #34 (Structure Phase 9 self-audit prompts and label gap issues): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Extends the rubric-over-free-form pattern from Phase 4 (self-review) to Phase 9 (self-audit), with a 6-item structured checklist and a 5-label taxonomy for filed gap issues.
- PR #54 (Apply consolidated template-rule backlog, #44/#46/#47): shipped 2026-06-14. Observed effect: pending — needs N cycles of data. Sharpens the external-anchor definition (#44: UNVERIFIED handling when the anchor can't run; #46: green CI is not verification when CI's scope excludes the changed files) and grounds the Tests-fix rubric item empirically (#47: stash-and-run FAIL→PASS, not scored from reasoning alone) — directly targeting the self-preference failure mode this finding's evidence documents for judgment-based rubric scoring.
- PR #57 (Fix documentation drift across the three sources of truth): shipped 2026-07-25. Observed effect: pending — needs N cycles of data. Restores `{{EXTERNAL_SIGNAL_LABEL}}` in the Phase 1 carried-over-PR bullet, where PR #54 had reintroduced a hardcoded "CI must be green first" after PR #28 generalized the anchor. An anchor named `CI` in a repo that has no CI is an anchor the loop cannot check, which degrades the self-review to exactly the unanchored critique this finding shows is neutral-to-harmful.
Expand All @@ -67,7 +67,7 @@ Last reviewed: 2026-07-25.
- The generated skill should include explicit human off-ramps: conditions under which the agent must stop and request review rather than auto-merge.

**Implementations.**
- PR #29 (Bias triage and batching toward higher-merge-rate issue types): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds a Phase 2 tiebreaker preferring documentation → CI/build → small refactors → bug fixes → performance, with stronger weighting in early cycles. Targets the merge-rate gradient by issue type; does not yet address the LOC/files ceilings (issue #17 is the structural-gates follow-up).
- PR #29 (Bias triage and batching toward higher-merge-rate issue types): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds a Phase 2 tiebreaker preferring documentation → CI/build → small refactors → bug fixes → performance, with stronger weighting in early cycles. Targets the merge-rate gradient by issue type; the LOC/files ceilings it left open (issue #17) shipped the same day in PR #30, recorded below.
- PR #30 (Hard PR scope gates): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds a Phase 3 ~400-net-LOC / ~10-files soft ceiling (~800 / ~20 hard stop) and a Phase 8 do-not-auto-merge path check (universal entries: `.github/workflows/*`, `security/`, >50-line deletions; plus a `{{DO_NOT_AUTO_MERGE}}` placeholder for repo-specific paths). Implements the first bullet of this finding's implications.
- PR #52 (Add documentation-accuracy and unit-test-expansion work-mode stages to Phase 2): shipped 2026-06-08. Observed effect: pending — needs N cycles of data. Promotes documentation-accuracy sweeps (Stage A) and unit-test expansion (Stage B) to first-class cycle work-modes, so a cycle with a thin or blocked issue backlog leans into the merge-rate gradient this finding documents instead of reaching for speculative feature work.
- PR #54 (Apply consolidated template-rule backlog, #49): shipped 2026-06-14. Observed effect: pending — needs N cycles of data. Refines the Phase 3 scope ceiling to count non-test net LOC and to permit dependency-coupled batches that can't be split without leaving an unused component, so the ceiling doesn't force an artificial rescope on PRs whose size is driven by test coverage or an inseparable dependency rather than actual scope creep.
Expand Down
Loading