//tools:stamped_tags builds the image tag from BUILD_SCM_TIMESTAMP and BUILD_SCM_REVISION:
stamp_substitutions = {"_TAG_": "{{BUILD_SCM_TIMESTAMP}}-{{BUILD_SCM_REVISION}}"},Neither key carries the STABLE_ prefix, so Bazel files them in volatile-status.txt and reports an unchanging digest for that file whatever it contains. The expand_template action's key never changes, so it is never re-run and _stamped.tags.txt is generated once and reused indefinitely.
On a machine whose Bazel cache persists between builds, every later build — of a different commit — pushes its image to the tag produced by the first build on that machine, silently overwriting what was published there before.
That is why it goes unnoticed here: the build_and_test job in .github/workflows/main.yaml, which runs the *_container_push steps, is runs-on: ubuntu-latest. GitHub-hosted runners are discarded after every job, so the cache never survives to serve a stale tag and the action re-executes every run. It only becomes visible on a self-hosted runner that keeps its cache between builds.
This affects every image built through container_push_official, so it is not just this repo: bb-remote-execution, bb-remote-asset and bb-portal each carry an identical tools/workspace-status.sh and consume this template via --override_module, and all of them stamp the same way.
Observed while building bb-portal on a long-lived self-hosted runner, all three facts in a single job:
git HEAD : 0250440 (2026-05-20) <- checkout correct
volatile-status.txt : BUILD_SCM_REVISION 0250440 <- status fresh and correct
_stamped.tags.txt : 20260407T093634Z-220cdb7 <- stale
-r-xr-xr-x 24 Apr 7 18:12 .../_stamped.tags.txt <- untouched for four months
Two pin bumps in between each published new image content to that April tag.
To be precise about the Bazel behaviour: volatile values are not permanently frozen — if the action is dirtied for another reason it re-runs and reads current values. Staleness persists only while nothing else about the action changes, which is the steady state here: tools/BUILD.bazel and the aspect_bazel_lib version both sat unchanged across those four months.
Suggested fix
Prefix the keys, in tools/workspace-status.sh and in the template together:
echo "STABLE_BUILD_SCM_REVISION $(git rev-parse --short HEAD)"
stamp_substitutions = {"_TAG_": "{{STABLE_BUILD_SCM_TIMESTAMP}}-{{STABLE_BUILD_SCM_REVISION}}"},The volatile bucket is for values that change every build, like the clock. A commit SHA is the opposite, so STABLE_ looks correct here on its own merits.
Script and template must be renamed together, or the placeholder matches nothing — and since the other bb repos ship their own workspace-status.sh against this template, they need the same change.
We work around it by deleting _stamped.tags.txt before each push. Happy to send PRs if useful.
//tools:stamped_tagsbuilds the image tag fromBUILD_SCM_TIMESTAMPandBUILD_SCM_REVISION:Neither key carries the
STABLE_prefix, so Bazel files them involatile-status.txtand reports an unchanging digest for that file whatever it contains. Theexpand_templateaction's key never changes, so it is never re-run and_stamped.tags.txtis generated once and reused indefinitely.On a machine whose Bazel cache persists between builds, every later build — of a different commit — pushes its image to the tag produced by the first build on that machine, silently overwriting what was published there before.
That is why it goes unnoticed here: the
build_and_testjob in.github/workflows/main.yaml, which runs the*_container_pushsteps, isruns-on: ubuntu-latest. GitHub-hosted runners are discarded after every job, so the cache never survives to serve a stale tag and the action re-executes every run. It only becomes visible on a self-hosted runner that keeps its cache between builds.This affects every image built through
container_push_official, so it is not just this repo: bb-remote-execution, bb-remote-asset and bb-portal each carry an identicaltools/workspace-status.shand consume this template via--override_module, and all of them stamp the same way.Observed while building bb-portal on a long-lived self-hosted runner, all three facts in a single job:
Two pin bumps in between each published new image content to that April tag.
To be precise about the Bazel behaviour: volatile values are not permanently frozen — if the action is dirtied for another reason it re-runs and reads current values. Staleness persists only while nothing else about the action changes, which is the steady state here:
tools/BUILD.bazeland theaspect_bazel_libversion both sat unchanged across those four months.Suggested fix
Prefix the keys, in
tools/workspace-status.shand in the template together:The volatile bucket is for values that change every build, like the clock. A commit SHA is the opposite, so
STABLE_looks correct here on its own merits.Script and template must be renamed together, or the placeholder matches nothing — and since the other bb repos ship their own
workspace-status.shagainst this template, they need the same change.We work around it by deleting
_stamped.tags.txtbefore each push. Happy to send PRs if useful.