Skip to content

[#14151] improvement(ci): Stop Fork CI on the first failed job and report results faster - #14152

Merged
yuqi1129 merged 3 commits into
apache:mainfrom
yuqi1129:fork-ci-fail-fast
Oct 23, 2026
Merged

yuqi1129 merged 3 commits into
apache:mainfrom
yuqi1129:fork-ci-fail-fast

Conversation

@yuqi1129

@yuqi1129 yuqi1129 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

  • fork-ci.yml: add a Stop on first failure job that polls the run's jobs once a minute, logs progress (19/28 jobs completed, 1 failed; waiting for ...), and cancels the run as soon as any job fails or times out. It is the only job with actions: write, runs only trusted script code (no PR build or tests), and uses ubuntu-latest because ubuntu-slim stops jobs after 15 minutes.
  • fork-ci-status.cjs: a run that ended because a job failed is reported as failure with the failed job in the check title (e.g. Failed: backend / BackendIT ...), instead of cancelled.
  • update-fork-ci.cjs: scale the scheduled sweep budget with the hourly quota (limit / 24, at least 60 calls), and keep a scheduled sweep polling for 11 minutes, one pass every two minutes (repositories with small quotas keep a single pass). Fork runs cannot notify upstream, and GitHub fires the five-minute schedule only every 10-20 minutes on apache/gravitino, so results used to appear 15-40 minutes after the fork run finished.
  • asf-allowlist-trusted.yml, conflict-marker-check.yml: run on ubuntu-slim. On the shared ubuntu-latest pool these few-second jobs queued for more than three hours, leaving the required ASF Allowlist Check pending. The allowlist action installs its own Python and supports ubuntu-slim.
  • fork-ci-status.cjs / update-fork-ci.cjs: when the run listing returns no fork run, show a waiting check (Waiting for Fork CI) instead of action_required "Enable Actions in the fork", and never replace a run already recorded on the check with an empty listing. During the current GitHub Actions incident, listWorkflowRuns intermittently returned 0 runs for existing runs, which turned reopened PRs into "Enable Actions".
  • CONTRIBUTING: describe the new re-run behavior, and that closing and reopening a PR refreshes the check but cannot re-run CI in the fork.

Why are the changes needed?

GitHub only allows "Re-run failed jobs" after the whole run completes, and Fork CI runs all suites in one run, so a single early failure meant waiting about two hours for the slowest suite before retrying. The PR check also stayed "in progress" for that time. With 60 calls per sweep, a finished fork run could take 30+ minutes to appear on apache PRs.

Fix: #14151

Does this PR introduce any user-facing change?

Contributors: one failed job now stops the rest of the Fork CI run; "Re-run failed jobs" is available immediately and also re-runs the jobs that were stopped.

How was this patch tested?

  • 41 Node tests in .github/scripts/tests/fork-ci.test.cjs (new: cancel on first failure, watcher ignores itself and exits after all jobs complete, failure title, single actions: write job on ubuntu-latest, quota-scaled sweep budget, waiting on a missing run, keeping a recorded run, multi-pass sweeps that mirror a run finishing within the window); actionlint.
  • Sandbox (a fork standing in for apache/gravitino, contributor fork with a forced allowlist failure and a long stand-in job): the failure cancelled the run 12 seconds later, the upstream check showed failure | Failed: asf_allowlist / asf-allowlist-check, and "Re-run failed jobs" was available immediately and re-ran the cancelled jobs (Required CI succeeded). A run with an 18-minute job passed with the watcher exiting on its own, and a real test failure in build / changes cancelled a run within one poll, with the progress lines in the watcher log.
  • Sandbox polling sweep (5 passes in one job, 00:41-00:51): a fork run that failed at 00:43:09 showed Fork CI=failure at 00:43:43, and one that passed at 00:49:14 showed success at 00:50:40, with no other event. On ubuntu-slim, Conflict Marker Check and ASF Allowlist Check reported 40-70 seconds after the push.
  • Sandbox: after close/reopen, a PR that had been wrongly marked "Enable Actions" showed Waiting for Fork CI and then the fork run once it was listed again.
  • This PR changes .github/workflows, so its own Fork CI check needs a trusted run dispatched by a committer.

…and refresh PR checks faster

- Add a Stop on first failure job to fork-ci.yml that logs progress each
  minute and cancels the run once any job fails or times out, so authors
  can re-run failed jobs immediately instead of waiting for the slowest
  suite. It is the only job with actions: write and runs on ubuntu-latest
  because ubuntu-slim stops jobs after 15 minutes.
- Report a run stopped this way as failure with the failed job in the
  check title.
- Scale the scheduled sweep budget with the hourly API quota.
Copilot AI balanced review requested due to automatic review settings October 22, 2026 15:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@yuqi1129

Copy link
Copy Markdown
Contributor Author

@yuqi1129 yuqi1129 closed this Oct 22, 2026
@yuqi1129 yuqi1129 reopened this Oct 22, 2026
…n a fork run is not listed

Run listings can miss existing runs, e.g. during GitHub incidents. Show a
waiting check instead of asking the author to enable Actions, and never
replace a run already recorded on the check with an empty listing. Note in
CONTRIBUTING that reopening a PR refreshes the check but does not re-run
Fork CI.
…d keep light checks off the shared pool

- Scheduled status sweeps keep polling for 11 minutes, one pass every two
  minutes, because GitHub fires five-minute schedules only every 10-20
  minutes on apache/gravitino and fork runs cannot notify upstream. Small
  quotas keep a single pass.
- Run Trusted ASF Allowlist and Conflict Marker Check on ubuntu-slim: these
  small jobs queued for over three hours on the shared ubuntu-latest pool.
@yuqi1129 yuqi1129 changed the title [#14151] improvement(ci): Stop Fork CI on the first failed job and refresh PR checks faster [#14151] improvement(ci): Stop Fork CI on the first failed job and report results faster Oct 23, 2026
jerryshao
jerryshao previously approved these changes Oct 23, 2026
@yuqi1129 yuqi1129 changed the title [#14151] improvement(ci): Stop Fork CI on the first failed job and report results faster [#14151] improvement(ci): Stop Fork CI on the first failed job Oct 23, 2026
@yuqi1129 yuqi1129 changed the title [#14151] improvement(ci): Stop Fork CI on the first failed job [#14151] improvement(ci): Stop Fork CI on the first failed job and report results faster Oct 23, 2026
@yuqi1129
yuqi1129 merged commit 8b2b30d into apache:main Oct 23, 2026
8 of 9 checks passed
@yuqi1129
yuqi1129 deleted the fork-ci-fail-fast branch October 23, 2026 01:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Improvement] Fork CI: retrying a failed job waits for the whole run, and PR checks refresh slowly

4 participants