IONOS(ci): migrate operational improvements from ncw-server sister project#248
Merged
printminion-co merged 23 commits intoMay 13, 2026
Merged
Conversation
Adds a concurrency block that cancels in-progress workflow runs when a new run is queued on the same PR or non-critical branch. Runs on ionos-dev, ionos-stable, and rc/* are keyed by run_id so concurrent pushes to release branches don't cancel each other. Mirrors the strategy used by the sister ncw-server build-artifact workflow (~/develop/easycloud/ncw-server/.github/workflows/build-artifact.yml).
Adds a "Check configuration" step that writes a markdown table to
$GITHUB_STEP_SUMMARY summarizing whether the trigger-remote-dev-worflow
job will fire — DISABLE_REMOTE_TRIGGER value, event type, branch, and
the predicted go/no-go. Purely observational.
Makes the most common build-pipeline question ("why didn't the GitLab
job get triggered?") visible at the top of the run summary, instead of
forcing a dig through skipped-job conditions.
Lists all GitHub Actions caches available to the run before detect-app-cache.sh consults them. Makes cache-miss investigations tractable by surfacing the actual cache key list in the run log instead of requiring 'gh cache list' from a developer machine.
Adds a 'Check prerequisites' step at the top of upload-to-artifactory that fails fast with ::error:: annotations if any of JF_ARTIFACTORY_URL, JF_ARTIFACTORY_USER, or JF_ACCESS_TOKEN are unset, instead of failing deep inside jf rt upload with a less obvious error. Wraps the jf rt upload call in a 3-attempt retry loop with exponential backoff (10s, 20s) to absorb transient Artifactory hiccups that otherwise red-flag the whole pipeline.
After a successful jf rt upload in the per-app matrix step, runs jf rt s against the target path to confirm the artifact is searchable. Log-only — a verification miss prints a warning but does not fail the build, since the upload itself already returned success. Surfaces the rare case where JFrog accepts an upload but the artifact is not immediately searchable (typically a routing/indexing issue on the Artifactory side).
Composite action that fetches the GitHub Actions job HTML URL via the REST API and exposes it as an output. Used by subsequent steps to attach job.html_url to JFrog artifact properties for traceability between an Artifactory artifact and the run/job that produced it. Uses a temp netrc file to keep the GH token off the curl command line, and trap-cleans both the netrc and the curl-error temp file on exit. Returns an empty output (not a failure) when the API call or job lookup fails, so callers can include it with continue-on-error: true. Copied from sister ncw-server (.github/actions/get-job-data/action.yml).
Converts the two jf rt upload --target-props sites (build-apps, upload-to-artifactory) from a single inline semicolon string to a bash array assembled per upload. Adds an invocation of the get-job-data composite action before each upload and conditionally appends job.html_url to the props array. Result: every artifact in JFrog now carries a clickable link back to the exact GitHub Actions job that produced it, simplifying post-incident traceability. Adds a Checkout step to upload-to-artifactory (self-hosted runner) since the local composite action requires the repo on disk.
The hidrive-next-build final job's setup-node was previously unconditionally caching npm dependencies, even when the user triggered workflow_dispatch with force_rebuild=true. That meant force_rebuild busted the JFrog and custom-npms caches but silently kept a stale npm cache. Drives the cache flag from the force_rebuild input: empty string when forcing, 'npm' otherwise. Matches the existing per-app composer cache gate (lines 478-485) so force_rebuild now uniformly clears every cache the workflow controls.
Both setup-php invocations (per-app build and final hidrive-next-build) were relying on shivammathur/setup-php's default, which can drift with upstream changes to the action. Pinning to 8.3 makes the runtime explicit and keeps test/build environments aligned with production. Aligns with the sister ncw-server build-artifact workflow.
Each successful matrix job in build-apps now appends a small markdown block to $GITHUB_STEP_SUMMARY listing the app, its SHA, path, and success status. With 7 parallel matrix jobs the run summary becomes a single-glance view of which apps actually built (vs. restored from cache) — useful when investigating cache-hit ratios. Aligns with the sister ncw-server build-artifact workflow.
The hidrive-next-build restore step previously only validated that the target directory existed after tar extraction. A corrupt or truncated archive could leave a near-empty directory and still pass this check, causing downstream build failures with confusing errors. Adds an appinfo/info.xml file check after both restore paths (cached apps and newly built apps). The check is conditional on APP_PATH starting with 'apps-' so themes/ submodules, which don't ship appinfo/info.xml, continue to pass with just the directory check.
Previously the script would exit 1 on the first app whose submodule
directory was missing or not a git repo, killing the whole pipeline.
This is overly strict: a partial submodule init (e.g. a network blip
during checkout) shouldn't block builds for the other six apps.
Now both edge cases queue the app for build with sha="unknown" and
log a non-fatal '⊘' marker to the step summary, then continue.
The resulting cache_key is non-collidable with normal runs ('-unknown'
suffix), so this never poisons the regular per-SHA cache. The
build itself proceeds because the matrix step doesn't depend on the
SHA being valid — only the cache key does.
Aligns with the sister ncw-server detect-app-cache.sh behavior.
Adds a parallel actions/upload-artifact step after each app build, making the freshly built tree downloadable from the workflow run page for one day. Complements the JFrog upload (which is the primary delivery channel) by giving in-flight access without an Artifactory login. retention-days: 1 keeps storage costs negligible — these are debugging conveniences, not durable artifacts.
…kflow Fixes the longstanding typo in the job key. Updates the job declaration and all references in step output (preflight summary written from prepare-matrix). Coordination notes: - No needs:/if: references the old name (the job is a leaf in the dependency graph — only the misspelled key itself). - If branch protection on ionos-dev/ionos-stable has a required status check named 'trigger-remote-dev-worflow', that check must be re-added under the new name in Settings → Branches. - No external webhook listens for this job name (verified by repo search; the GitLab trigger is invoked from within the job, not the other direction).
v5 drops Node.js 16 in favor of Node 20 (which is what GitHub-hosted runners ship today) and includes upstream fixes for partial clone behavior. All checkout invocations updated to the same pinned SHA. Aligns with the sister ncw-server build-artifact workflow (which uses @v5).
v6 drops Node 16 support and adds compatibility with Node 24, plus the latest npm cache and proxy fixes. All setup-node invocations updated to the same pinned SHA. Aligns with the sister ncw-server build-artifact workflow (which uses @v6).
Switches from the unpinned @v4 short tag to SHA-pinned v5.0.0, matching the project's policy of full-SHA pinning for security review traceability. v5 brings significantly improved download performance (parallel streaming) for large workflow artifacts. Aligns with the sister ncw-server build-artifact workflow (which uses @v5).
Brings in upstream fixes for credential handling on self-hosted runners. Minor v3.x bump; no API changes. Adds a version comment alongside the SHA for human readability.
Latest v5.x. No API changes; routine bump bringing in OCI manifest label improvements. Adds version comment alongside the SHA.
Extends the push branch filter to fire the workflow for release-candidate
branches ('rc/**') and developer feature branches ('*/dev/*', e.g.
'mk/dev/foo'). Previously these branches only built via the
pull_request trigger, which is awkward for developers who want a
fast feedback loop on push before opening a PR.
Companion commits in this branch route those branches' artifacts to
distinct Artifactory paths and extend the remote-trigger conditions
to match.
Aligns with the sister ncw-server build-artifact workflow.
Extends upload-to-artifactory's gating condition to allow PR,
workflow_dispatch, ionos-dev/ionos-stable, rc/*, and */dev/* runs to
land in Artifactory.
Adds two new ARTIFACTORY_STAGE_PREFIX rules:
- rc/* → uses the literal branch name as prefix (e.g. 'rc/33.0')
- */dev/* → uses 'dev-<owner>' where <owner> is the slug before /dev/
with non-alphanumerics replaced by '-' (e.g. 'mk/dev/foo'
→ 'dev-mk')
This separates feature-branch and release-candidate artifacts from
the main dev/stable streams so they don't collide on the "latest"
NC_VERSION file name. The path table is documented inline next to
the derivation.
Extends three things to handle the newly-buildable rc/* and */dev/*
branches:
1. trigger-remote-dev-workflow's gating condition now accepts
rc/* and */dev/* refs in addition to ionos-dev/ionos-stable.
2. BUILD_TYPE mapping in the GitLab webhook payload:
- rc/* → 'rc'
- */dev/* → 'dev-<owner-slug>' (matches the Artifactory stage prefix
introduced in the companion 'route rc/*…' commit so the GitLab
side can reason about artifact location consistently).
3. The prepare-matrix preflight-summary branch-validity regex now
matches the same four patterns, so the predicted go/no-go in the
step summary stays accurate.
GitLab pipeline coordination: the new BUILD_TYPE values land in the
trigger payload's `variables[BUILD_TYPE]`. Coordinate with the GitLab
maintainer to confirm the downstream pipeline handles 'rc' and
'dev-*' explicitly (or accepts them as fallthrough to 'dev'-like
behavior).
Branch uploads (ionos-dev / ionos-stable / rc/* / */dev/*) now write to '<stage>/hidrive-next-<NC_VERSION>/<shortSha>/hidrive-next-<NC_VERSION>.zip' instead of overwriting a single '<stage>/hidrive-next-<NC_VERSION>.zip' slot. The nested layout preserves every build by sha and lets us debug a regression against the exact artifact a given commit produced. PR uploads are unchanged: 'dev/pr/hidrive-next-pr-<number>.zip' is still the per-PR canonical path, since PRs benefit from "overwrite the latest on the open PR" semantics.⚠️ Downstream consumer impact: - trigger-remote-dev-workflow passes ARTIFACTORY_LAST_BUILD_PATH as a literal full path to the GitLab pipeline, so GitLab consumers that re-derive the path from a glob (`dev/hidrive-next-*.zip`) will break. Coordinate with the GitLab pipeline maintainer before merging this commit — or revert if a "latest" symlink is needed. - The Artifactory cleanup policy may need adjustment: per-sha retention will grow the repo over time. The previous flat layout retained one slot per branch. Aligns with the sister ncw-server build-artifact workflow.
printminion-co
force-pushed
the
mk/dev/ci-sister-migration
branch
from
08e4aab to
076a365
Compare
printminion-co
merged commit f20746f
into
mk/dev/cached-matrix-build
15 of 24 checks passed
There was a problem hiding this comment.
Pull request overview
This PR migrates CI operational improvements into the HiDrive Next build workflow, expanding branch coverage, adding traceability/observability, and improving artifact upload/restore behavior.
Changes:
- Adds workflow concurrency, cache/debug summaries, branch routing, retry logic, and JFrog upload metadata.
- Introduces a local composite action to fetch GitHub Actions job URLs for artifact traceability.
- Updates cache detection behavior and restore validation for app artifacts.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
.github/workflows/hidrive-next-build.yml |
Expands CI triggers, updates action pins, adds Artifactory upload/traceability improvements, restore validation, and remote trigger updates. |
.github/scripts/detect-app-cache.sh |
Changes missing/non-git app handling to continue and schedule a rebuild instead of failing immediately. |
.github/actions/get-job-data/action.yml |
Adds a composite action that queries the GitHub Actions API for a job HTML URL. |
Comments suppressed due to low confidence (1)
.github/workflows/hidrive-next-build.yml:577
- The same
apps-*pattern misses apps restored under the standardapps/directory, so newly built app archives fromapps/are accepted without verifyingappinfo/info.xml. Matchapps/*as well as theapps-*directories to keep the restore validation consistent with the intended coverage.
# appinfo/info.xml lives in standard Nextcloud apps but not in themes/
case "$APP_PATH" in
apps-*)
if [ ! -f "$APP_PATH/appinfo/info.xml" ]; then
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+729
to
+737
| - name: Get Job data | ||
| id: get_job_data | ||
| continue-on-error: true | ||
| uses: ./.github/actions/get-job-data | ||
| with: | ||
| job-name: 'Push to artifactory' | ||
| github-token: ${{ github.token }} | ||
| repository: ${{ github.repository }} | ||
| run-id: ${{ github.run_id }} |
| fi | ||
| # appinfo/info.xml lives in standard Nextcloud apps but not in themes/ | ||
| case "$APP_PATH" in | ||
| apps-*) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrates 23 atomic CI improvements from the sister
ncw-serverproject'sbuild-artifact.ymlworkflow, plus a few in-repo cleanups discovered along the way. Stacked on #226 — base ismk/dev/cached-matrix-build. PR #226 should merge first.Each commit is intentionally atomic and individually revertable; commit messages explain rationale and any coordination notes.
Tiers (23 commits)
Tier 1 — concurrency, retry, observability
prepare-matrixpreflight step writes a remote-trigger go/no-go table to$GITHUB_STEP_SUMMARYgh cache listdebug step surfaces existing cache keys in the logupload-to-artifactorygains secrets preflight + 3-attempt retry with exponential backoff (10s, 20s)jf rt sverification probe after successTier 2 — traceability via
get-job-datacomposite action6. New
.github/actions/get-job-data/action.yml(copied from ncw-server) — fetches GitHub job HTML URL via REST7. All three
jf rt upload --target-propssites use bash-array assembly and embedjob.html_urlfor back-traceabilityTier 4 — small ergonomics
8.
force_rebuild=truenow also bypasses the npm cache in the final job9.
php-version: '8.3'explicitly pinned (was implicit)10. Per-app build completion summary in run page
11. Restored apps validated for
appinfo/info.xml(skipped forthemes/which doesn't ship one)Tier 6 — operational cleanups
12.
detect-app-cache.shcontinues on missing/non-git submodule instead ofexit 113. Per-app
upload-artifact@v4atretention-days: 1for in-flight debugging14. Renamed misspelled
trigger-remote-dev-worflow→trigger-remote-dev-workflow(see Coordination)Tier 5 — action version bumps, SHA-pinned
15.
actions/checkout→ v5.0.116.
actions/setup-node→ v6.4.017.
actions/download-artifact→ v5.0.0 (was unpinned@v4)18.
docker/login-action→ v3.7.019.
docker/metadata-action→ v5.10.0Tier 3 — broader branch coverage
20.
on.push.branchesnow includesrc/**and*/dev/*21.
upload-to-artifactoryroutes those branches to nested Artifactory prefixes (rc/<branch>,dev-<owner>)22.
trigger-remote-dev-workflowaccepts the new branches and mapsBUILD_TYPEaccordinglyArtifact path nesting
23. Branch uploads now nest by
hidrive-next-<NC_VERSION>/<shortSha>/(PRs unchanged) — preserves every build per SHACoordination required before merge
ARTIFACTORY_LAST_BUILD_PATHis now nested. The GitLab side accepts the path as a literal value so this should be transparent, but please confirm with the GitLab pipeline maintainer that no glob/derived path assumptions break.trigger-remote-dev-worflow(misspelled) is a required status check, re-add the corrected name in Settings → Branches.Test plan
<owner>/dev/<topic>branch → confirmupload-to-artifactoryruns, lands atdev-<owner>/hidrive-next-<ver>/<sha>/...zip, andtrigger-remote-dev-workflowfires withBUILD_TYPE=dev-<owner>prepare-matrixstep summary contains the "Remote Trigger Configuration" tableworkflow_dispatchwithforce_rebuild: true→ confirm all caches bypassed (custom-npms, per-app, npm, composer)app.name,app.sha,vcs.*,job.html_urlprops are present