ci: route refreshed visual baselines through a PR instead of a direct push#3285
Conversation
… push The develop branch ruleset rejects the final direct push in refresh-visual-baselines.yml (GH013: changes must be made through a pull request), failing every baseline refresh after the expensive rebuild + screenshot work already succeeded. Convert the final step to the PR + auto-merge pattern already used by the sibling refresh-packages-baseline.yml: commit the refreshed PNG(s) onto a throwaway chore/refresh-baseline-* branch, gh pr create against the dispatched branch, and gh pr merge --auto --squash --delete-branch (falling back to leaving the PR open if repo auto-merge is disabled). Adds pull-requests: write to the job permissions (gh pr create/merge fail silently on a contents: write-only token) and extracts the branch/commit/PR/auto-merge flow into an injection-safe helper, tools/gh-open-refresh-baseline-pr.sh, pinned by the new RefreshVisualBaselinesPrPushSpec. Applies the TDD-verified patch held in the wheels-bot comment on the issue (the bot's GitHub App lacks the workflows permission needed to push workflow-file changes itself). Fixes #3283 Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com>
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR — This PR fixes #3283: the Refresh visual baselines workflow used to git push origin "HEAD:$BRANCH" at the end, which the develop ruleset now rejects with GH013: Changes must be made through a pull request. It re-routes the refresh through a throwaway chore/refresh-baseline-* branch → gh pr create → gh pr merge --auto, extracted into tools/gh-open-refresh-baseline-pr.sh, grants the job pull-requests: write, and pins the invariant with a static regression spec. The change is well-scoped, TDD-verified (red 4-fail → green 4-pass), and mirrors the proven refresh-packages-baseline.yml pattern. Verdict: comment — no blocking issue; one operational caveat on the auto-merge step worth a maintainer's eyes before relying on it landing unattended.
Correctness
-
tools/gh-open-refresh-baseline-pr.sh:96— auto-merge may be enabled but never fire, because the refresh PR's required checks won't run. The PR is opened withGH_TOKEN: ${{ secrets.GITHUB_TOKEN }}, and GitHub's infinite-loop guard suppresses workflow triggers on GITHUB_TOKEN-authored PRs. The sibling workflow this PR mirrors documents exactly this and deliberately does not auto-merge because of it:.github/workflows/refresh-packages-baseline.yml: "CI gotcha: PRs opened with GITHUB_TOKEN don't trigger other workflow runs (GitHub's infinite-loop guard). Reviewers should eyeball the single-PNG diff."visual-regressionlives inweb-deploy.yml, which fires onpull_request→developforweb/**paths — the refresh PR touchesweb/tests/visual-baselines/*.png, so it matches the path filter but the run is suppressed by the token. If the target branch's protection requires that check,gh pr merge "$pr_url" --autoenables auto-merge but the PR then waits forever on a check that never runs. The helper'sif gh pr merge … ; then … else …only catches the case where the command itself errors (auto-merge disabled repo-wide), not the enabled-but-stuck case — so the "left open for manual merge" log line won't print, and a human has to notice the wedged PR on their own.This is non-blocking: the primary bug (the GH013 crash) is genuinely fixed, and the worst case degrades to the same manual-merge fallback the PR body already anticipates. But the stated goal ("auto-merges once the required checks pass … lands unattended") likely won't hold when the target branch has required checks. Consider either (a) documenting in the helper/step summary that a maintainer must merge manually if the refresh PR's checks don't auto-run under GITHUB_TOKEN, or (b) matching the sibling and dropping
--autoentirely, given both share the token limitation. At minimum, verify against a target branch with the real required-check set during the suggested liveworkflow_dispatchrun.
Cross-engine
No framework runtime code changed. The one CFML file added (RefreshVisualBaselinesPrPushSpec.cfc) is a static-content spec — fileRead + reFindNoCase only, ##3283 correctly escaped in string literals (## → literal #), // comments left un-escaped as allowed. No closures-as-named-args, no reserved-scope names, no tag wrappers. Extends wheels.WheelsTest with BDD describe/it. Clean.
Tests
vendor/wheels/tests/specs/cli/RefreshVisualBaselinesPrPushSpec.cfc pins the four invariants that matter — no residual git push … HEAD:<target> in either the workflow or the helper, pull-requests: write present, gh pr create + gh pr merge --auto present, and the workflow actually invoking the helper. Genuine TDD (red 4-fail → green 4-pass reported), and the spec asserts on the fix's observable contract rather than incidental text. Good coverage for a step whose live side effects (git + gh) can't be unit-exercised.
Commits
ci: route refreshed visual baselines through a PR instead of a direct push — valid type (ci), no scope, subject not ALL-CAPS, ≤100 chars. Conforms to commitlint.config.js.
Docs
- Changelog fragment
changelog.d/3283-refresh-baselines-pr-push.fixed.mduses a valid type (fixed) and complete bullet text — correct per the fragment system (no directCHANGELOG.mdedit). Minor: the commit itself isci:(internal workflow plumbing, not user-facing), so a changelog fragment is arguably optional here — but erring toward documenting is harmless, not a defect. - Workflow header comments and the step summary are updated to describe the new PR flow accurately. Nice.
…R fallback Addresses review findings on the ##3283 fix: - Restore the pre-##3283 direct push as an if-guarded fast path: dispatching on an unprotected branch (the header's documented feature-branch flow) lands the commit immediately again, instead of always detouring through a PR. A GH013 rejection now falls through to the PR fallback instead of failing the job. - Drop `gh pr merge --auto` on the fallback PR: it is authored by the workflow's GITHUB_TOKEN, and GitHub never fires pull_request workflows for GITHUB_TOKEN-authored PRs, so the required checks sit "Expected" forever and auto-merge wedges silently — the exact gotcha the sibling refresh-packages-baseline.yml documents and avoids the same way. - Honest operator messaging: the helper reports delivery=push|pr (+ pr_url) via GITHUB_OUTPUT and the step summary now tells the maintainer the fallback PR cannot land unattended (close/reopen to trigger CI, or review the PNG diff and merge). - Seed the throwaway branch name with RUN_ATTEMPT so a re-run of a failed job (same run_id) cannot collide non-fast-forward with attempt 1's leftover branch. - Harden the spec: strip whole-line comments before matching so only EXECUTABLE code satisfies it, require every direct push in the helper to be if-guarded, forbid `gh pr merge --auto`, and pin the RUN_ATTEMPT branch suffix. Verified red against both the original bug and the previous iteration; 5/5 green on Lucee 7 + SQLite, cli area 131 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com>
|
Addressed the reviewer's auto-merge caveat (and a few adjacent findings) in bba7eed:
|
There was a problem hiding this comment.
Wheels Bot — Reviewer
TL;DR — This PR fixes #3283: the Refresh visual baselines workflow used to git push origin "HEAD:$BRANCH" as its final step, which the develop ruleset rejects with GH013: Changes must be made through a pull request, wasting the whole rebuild+screenshot run. Delivery now lives in tools/gh-open-refresh-baseline-pr.sh: the direct push is kept as an if-guarded fast path (unprotected feature branches, the documented dispatch flow), and a ruleset rejection falls through to a throwaway chore/refresh-baseline-* branch + gh pr create left for a maintainer. The job gains pull-requests: write, and a static regression spec pins the invariants. This head (bba7eed4) also resolves my prior review's only concern by dropping the inert gh pr merge --auto step. The change is well-scoped, TDD-verified, injection-safe, and mirrors the proven refresh-packages-baseline.yml pattern. Verdict: approve.
Resolved from prior review (465d4406)
- Auto-merge that could never fire — fixed. My earlier comment flagged that
gh pr merge … --autoon aGITHUB_TOKEN-authored PR would wedge, since GitHub suppressespull_requesttriggers on such PRs so required checks never run. The commitci: keep guarded direct baseline push, drop inert auto-merge on the PR fallbackremoves it entirely — verified: a grep forgh pr merge/--autoovertools/gh-open-refresh-baseline-pr.shand.github/workflows/refresh-visual-baselines.ymlreturns nothing, and the spec now actively pins its absence (RefreshVisualBaselinesPrPushSpec.cfc:120asserts the combined helper+workflow contains nogh pr merge … --auto). The fallback PR is now correctly left for a maintainer, matching the sibling workflow.
Correctness
Clean. The fast path if git push origin "HEAD:${TARGET_BRANCH}"; then … exit 0; fi (gh-open-refresh-baseline-pr.sh:100) is inside an if condition, so under set -euo pipefail a rejection does not trip errexit — it falls through to the PR path as intended. RUN_ATTEMPT is folded into the throwaway branch name (chore/refresh-baseline-${SITES}-${RUN_ID}-${RUN_ATTEMPT}) so a re-run of a failed job does not collide non-fast-forward with the branch left by attempt 1 — a real edge case, correctly handled and spec-pinned. The step-summary branches (push / pr / changed-but-undelivered / no-drift) cover all delivery output states.
Security
No injection surface. All inputs arrive via env and reach git/gh as single argv elements (--title, --body-file, never a shell-expanded command string); SITES is constrained to the workflow choice list and RUN_ID/RUN_ATTEMPT are numeric. The one word-split (git add -- $ADD_PATHS, gh-open-refresh-baseline-pr.sh:74) is intentional, shellcheck-annotated, and workflow-controlled. pull-requests: write is scoped to this single job; contents: write is unchanged.
Cross-engine
No framework runtime code changed. The one CFML file (RefreshVisualBaselinesPrPushSpec.cfc) is a static-content spec — fileRead + reFindNoCase/reFind only, issue number correctly escaped in string literals, the stripCommentLines closure assigned to a var before use (not passed as a constructor named arg). Extends wheels.WheelsTest with BDD describe/it. Reported green on Lucee 7 + SQLite. Clean.
Tests
vendor/wheels/tests/specs/cli/RefreshVisualBaselinesPrPushSpec.cfc follows the established static-check prior art (BotResolveConflictsLoopSafeSpec, expandPath("/wheels/../..")) and pins the invariants that matter: no git push in the workflow, the workflow invokes the helper, pull-requests: write present, every helper git push … HEAD: is if-guarded, an executable gh pr create exists, no auto-merge, and the branch name carries both RUN_ID and RUN_ATTEMPT. It strips comment lines first so a comment mentioning gh pr create cannot satisfy the assertions — the right rigor for a step whose live git/gh side effects cannot be unit-exercised. Genuine red→green TDD reported.
Commits
Both conform to commitlint.config.js: valid ci type, no scope, subjects not ALL-CAPS and ≤100 chars, messages explain the "why."
Docs (non-blocking nits)
- Stale PR description. The Summary and Extra-scrutiny sections still describe
gh pr merge --auto --squash --delete-branchand "The helper enables auto-merge on the baseline-refresh PRs it opens" — behavior removed in the final commit. The shipped helper opens the PR and leaves it for a maintainer with no auto-merge (and the spec now forbids auto-merge). Worth updating the description so a maintainer reading it is not misled; no code impact. ci:change with a changelog fragment.changelog.d/3283-refresh-baselines-pr-push.fixed.mdis well-formed (fixedtype, complete bullet), but this is internal workflow plumbing dispatched only by maintainers, so a user-facing fragment is arguably optional here. Harmless — erring toward documenting is fine.
Closes #3283
Summary
Dispatching Actions → Refresh visual baselines completes the expensive rebuild + screenshot +
--updatework, then dies at the final step: the workflow pushes the refreshed PNG(s) straight back to the dispatched branch (git push origin "HEAD:$BRANCH"), which thedevelopbranch ruleset now rejects withGH013: Changes must be made through a pull request.This converts the final step to the PR + auto-merge pattern already proven in the sibling
refresh-packages-baseline.yml:.github/workflows/refresh-visual-baselines.yml— the "Commit and push refreshed baselines" step becomes "Open refresh PR and enable auto-merge": it now invokes the new helper instead of pushing directly, and the jobpermissions:block gainspull-requests: write(acontents: write-only token fails thegh prcalls silently). Doc comments and the step summary updated to describe the PR flow.tools/gh-open-refresh-baseline-pr.sh(new) — commits the refreshed baseline(s) onto a throwawaychore/refresh-baseline-<sites>-<run_id>branch,gh pr createagainst the dispatched branch with a conventional, commitlint-safe title (chore(web): refresh visual baseline(s) (<sites>), ≤100 chars), thengh pr merge --auto --squash --delete-branch. If repo auto-merge is disabled it logs and leaves the PR open for a manual merge instead of failing the job. All inputs arrive viaenv(never spliced into a shell string), preserving the workflow's Actions-injection-safe posture. Passes shellcheck.vendor/wheels/tests/specs/cli/RefreshVisualBaselinesPrPushSpec.cfc(new) — static-content regression spec (same pattern asBotResolveConflictsLoopSafeSpec) pinning: nogit push … HEAD:<target>in the workflow or helper,pull-requests: writepresent,gh pr create+gh pr merge --autopresent, and the workflow actually invoking the helper.changelog.d/3283-refresh-baselines-pr-push.fixed.md— changelog fragment.This applies the TDD-verified patch the wheels-bot held in its comment on #3283 — the bot's GitHub App lacks the
workflowspermission required to push changes to workflow files, so it could not open this PR itself. Pushed here with human credentials.Test plan / status
TDD red → green, verified against a Lucee 7 + SQLite server serving this exact working tree:
wheels.tests.specs.cli.RefreshVisualBaselinesPrPushSpec→ 0 pass / 4 fail / 0 error, each failure citing the intended invariant (direct push present, missingpull-requests: write, no PR flow, missing helper).cliarea (directory=wheels.tests.specs.cli): 21 bundles, 131 pass, 0 fail, 0 error — no regressions.shellcheckandbash -n.Extra-scrutiny areas (sensitive: CI workflow change)
.github/workflows/— the changed step only runs on manualworkflow_dispatch, and the job token gainspull-requests: write(scoped to this job only;contents: writeis unchanged).refresh-packages-baseline.yml); required checks on the target branch still gate the actual merge. Repo auto-merge is currently known to toggle off after some admin PATCH operations — the helper degrades gracefully to leaving the PR open.workflow_dispatchrun; suggest dispatching Refresh visual baselines (sites=guides) once merged to confirm the PR flow (the runner'sgh/GITHUB_TOKENpath can only be proven live).Cross-engine notes
No framework runtime code changed. The only CFML added is a static file-content spec (string/regex/
fileReadonly,##correctly escaped in strings) — no closures-as-named-args, no reserved-scope names, no tag wrappers. Compiles and runs green on Lucee 7; nothing engine-specific in the spec surface.🤖 Generated with Claude Code