Two version streams are minted by hand, one automatically; everything else is CI-owned. Never create a version tag or move a channel tag by hand.
| Stream | Scheme | Minted by | Reaches the cluster |
|---|---|---|---|
| Platform (the core Flux roots) | CalVer YYYY.M.SEQ (2026.7.8) |
Release platform workflow | next core Flux reconcile (~1 min) |
purdue-af image (Dockerfile, pixi/base) |
semver v0.X.Y (v0.13.0) |
Release image workflow | default profile at the next platform release; the pre-release profile pulls :pre-release on every spawn |
| agentic-interface image | semver agentic-interface-vX.Y.Z |
ci.yml publish stage — automatic |
same pipeline — experimental Flux reconcile (~1 min); its Deployment pins the released tag, never :latest |
Experimental Flux source (main-validated) |
moving branch | ci.yml publish stage, behind ci-ok |
experimental Flux reconcile (~1 min) |
Continuous (:latest, :pre-release, sha-) |
moving tags | ci.yml publish stage, behind ci-ok |
on pod restart / session spawn |
:latest-only images |
:latest |
ci.yml publish stage, behind ci-ok — no release step |
af-pod-monitor at the next spawn; af-node-monitor, supersonic-model-manager, agentic-analysis and ray-nrp on pod restart; a node keeps the af-node-monitor it first pulled |
| interLink plugin images | upstream ref :$PLUGIN_REF |
a PLUGIN_REF bump |
when the interLink values pin the new ref (slurm, kubernetes) |
Global env (pixi/global) |
the purdue-af image version | Release image workflow, as a copy in apps/af-utils/pixi-global-sync/release/ |
with the hub's new default image, at the next platform release; pixi-global-sync installs it into /work/pixi/global |
| Upstream sources | branches and moving tags outside this repo | upstream commits | next reconcile or pod restart, with no commit here: krab's chart and code location image, servicex-eos's fork chart and its :local images. No rollback below applies |
Every commit runs one pipeline (ci.yml) and
nothing is published unless every stage passed for that exact commit. The
publish stage runs on main only; the two releases are workflow_dispatch
runs from the Actions tab.
A delivery that fails leaves the workloads running as they were, and the status
badges read git, not the cluster. AFFluxRootNotReady and
AFHelmReleaseNotReady report it, and
kubectl -n cms get gitrepositories,kustomizations,helmreleases shows the cause
in its READY and SUSPENDED columns.
flowchart TD
push(["push/PR into main branch"]) --> checks["CI: syntax, pixi, images, e2e"]
checks --> ok{{"ci-ok?"}}
ok -->|no| stop(["nothing published"])
ok -->|yes| passed["ci passed"]
passed --> pub["publish"]
pub --> tags["pre-release / latest docker tags"]
pub --> mv["advance main-validated branch"]
mv --> exp["Flux: reconcile experimental components"]
passed -.->|manual run by admin| img["Release image"]
passed -.->|manual run by admin| plat["Release platform"]
img -.-> plat
plat -.-> core["Flux: reconcile core components"]
Mint one whenever core components need to reach the cluster: hub
configuration, monitoring, cronjobs, manifest changes. (Experimental
components track main-validated and never need one.)
Actions → Release platform → Run workflow computes the next
YYYY.M.SEQ, tags it, and publishes a GitHub Release with generated notes —
no file edits, no image tags. Core Flux advances within ~1 minute. It refuses
a commit whose ci-ok has not passed, a running pipeline included, unless
force is set; a Release image commit has no pipeline and passes on its
parent's gate.
Rollback: delete the newest Release together with its tag. Flux tracks the newest platform git tag, so it re-resolves to the previous one and re-applies that commit on the next reconcile.
gh release delete 2026.7.9 --cleanup-tag # --cleanup-tag is what matters
Deleting the Release object alone rolls back nothing. And this rolls back
what is deployed, not history: main still contains the offending commits —
fix or revert them, or the next tag re-ships them.
Release when the content soaking as :pre-release should become the default
session environment. major — never, until the AF moves from R&D to
Operations; minor — breaking changes for users, or a major change to the
codebase; patch — anything else.
Preconditions, enforced by the workflow (bypass only with force): ci-ok
green on main HEAD, and the :pre-release digest identical to the image of
the current repo state. Complete the manual checklist in
docker/purdue-af/README.md first.
- Actions → Release image → Run workflow — choose the bump (or an
explicit
version). It adds the semver tag to the same digest that passed CI (never a rebuild), rewrites every version spot in the hub's values.yaml, the pixi-global-sync image and the workspaceversionofpixi/baseandpixi/global, and copies the stampedpixi/globalmanifest and lock toapps/af-utils/pixi-global-sync/release/, the env pixi-global-sync installs (bump-af-version.py, count-verified). It commits tomainand tagsv<version>. A second commit then sets the two workspace versions inpixi/to<version>+dev, so no other commit there carries a released version; both commits and the tag go in one atomic push, and the workflow publishes a Release. - Actions → Release platform → Run workflow — always the second step:
the bump commit reaches the hub, pixi-global-sync and
/work/pixi/globalonly once a platform tag covers it.
A workspace version is not an image input (image-inputs.sh hashes a
pixi.toml without it), so neither commit rebuilds an image or re-runs a
memoized pixi check.
Rollback: git revert both Release image commits on main
(git revert <+dev commit> <release commit>), then mint a new platform tag;
the hub and /work/pixi/global go back together. Never delete a v* tag —
the pins live in a values.yaml commit, so deleting the tag rolls back
nothing. Old semver tags stay on ghcr forever; the
registry GC never deletes release tags.
No button, no token. The publish stage of every green main pipeline checks
whether the digest it just validated (in-<hash> of the tested tree) is what
apps/agentic-interface/deployment.yaml
pins. If not, it mints the next version: promotes that exact digest to a bare
X.Y.Z tag on ghcr (never a rebuild), rewrites the Deployment pin
(bump-agentic-version.py, count-verified), pushes an auto-release commit
to main, tags agentic-interface-v<version>, and advances main-validated
to the bump commit in the same run — experimental Flux rolls the pod onto the
released tag about a minute later. Merge → deployed version in one pipeline,
with no human step.
The bump kind is chosen automatically: minor when the skill
(.claude/skills/purdue-af-agentic-interface — the user-facing contract of
the MCP server, and an input of the image hash) changed since the last
release, patch otherwise. So a skill-only change still mints and deploys a
new version, and the version number tells users whether their side of the
interface moved. The skill states the MAJOR.MINOR it ships in (its Skill version: line); the checks stage fails, naming the value to set, until that
stamp matches the bump, and the server announces it so an agent can tell its
user how far behind their copy is. Major versions are never minted
automatically; if one is ever wanted, --set it by hand in a normal commit
and let CI validate it.
The bump commit is the ONE deliberate exception to "everything on
main-validated ran its own pipeline": it is pushed with GITHUB_TOKEN
(which triggers no workflows), so instead it is
validated in-run — the step proves the commit differs from the tree this
pipeline just validated by exactly the one rewritten image-tag line, refuses
to release if main moved during the run, and stamps a ci-ok check run on
the bump commit via the API (annotated with the validating run) so the
purdue-af release gate keeps working. The bump commit changes only the
Deployment (not an image input), so the next pipeline sees digest == pin and
no-ops — the cycle terminates after one bump.
The server keeps no MCP sessions, so each auto-release is a rolling update: the new pod is ready before the old one stops, and a call in flight has the pod's termination grace period to finish. A multiple-choice question asked by the old pod and answered to the new one is refused, and the agent calls the tool again.
Rollback: git revert the CODE commit that produced the bad image — not
the auto-release commit (the digest check would just re-bump onto the same
bad image). The reverted tree hashes back to the previous good image, and the
next pipeline auto-releases a new patch version pointing at those good bytes.
Never delete an agentic-interface-v* tag.
- Channel tags (
:latest,:pre-release),sha-provenance tags andmain-validatedmove only in theci.ymlpublish stage, after every stage of the same commit is green.in-<hash>tags name build content, not a validated state: the images stage pushes them as soon as an image builds, on same-repo PRs too (docker/REGISTRY.md). Theprotect-main-validatedruleset blocks deleting the branch and leaves updates open, soGITHUB_TOKENcan push it. Publish moves it forward only: a re-run of a commit behind it fails, and after a history rewrite publish refuses until the branch is moved by its owner. - Workflows push with
GITHUB_TOKEN, whose pushes trigger no workflow:- the two Release image commits get no CI run and no
ci-ok. Gate 1 of the next image release fails until another commit lands onmainand passes CI, orforceis used. The platform tag of step 2 points at the+devcommit, which CI has not run on. - a lock commit from
ci-pixi-base.yml/ci-pixi-global.ymlgets its follow-up run fromretrigger-after-lock-push.sh: an approvedpull_requestrun, or else aworkflow_dispatchrun, which never publishes. - Release platform is unaffected: it pushes only a tag, and no workflow triggers on tags.
- the two Release image commits get no CI run and no