Skip to content

Docker tag cleanup - #274

Merged
ericmj merged 17 commits into
mainfrom
docker-tag-cleanup
Jul 30, 2026
Merged

ericmj merged 17 commits into
mainfrom
docker-tag-cleanup

Conversation

@ericmj

@ericmj ericmj commented Jul 1, 2026 •

Copy link
Copy Markdown
Member

Prunes hexpm/elixir-amd64 and hexpm/elixir-arm64, which hold the single-arch images the multi-arch manifests are assembled from rather than images anyone pulls directly. A tag built more than 30 days ago is removed and the multi-arch image it backs stays available. Requesting an image now reserves it from cleanup permanently, and reserves it without rebuilding when it already exists.

That is ~1.97M of the table's 3.08M rows. It also fixes manifest_mismatches/3, the diff the checker runs every 15 minutes for both erlang and elixir: measured against a prod-scale snapshot it drops from ~580ms warm (~3.4s cold, ~1.6GB of buffer reads) to ~9ms, because the plan stops hash-joining a million rows per side and becomes a nested loop over a few thousand.

What it deliberately does not do

The manifest repos are never pruned. The rule was going to delete hexpm/erlang and hexpm/elixir tags neither pulled nor built in 180 days, reading Docker Hub's tag_last_pulled. That field does not mean what the rule needed. Measured 2026-07-29: 99.5% of all 36,117 hexpm/erlang tags share the single pull date 2026-07-25 and nothing in the repo is more than five days idle; hexpm/elixir shows the same pattern spread over five days. Something walks every tag in both repos roughly every 80 days and resets the field. The threshold sat above the crawl interval, so the rule could never fire on disuse — the 0.33% that crossed 180 days were the tags the last crawl happened to miss, about 3,200 rows out of 3.08M, chosen close to arbitrarily and with the blast radius set by someone else's schedule. The last_pulled column, its migration and its index went with it.

The erlang per-arch repos are excluded. They look equally disposable and are not. DockerChecker.expected_elixir_tags/0 derives the entire Elixir build matrix from the erlang per-arch rows in the mirror, filtered by latest_erlang_versions/0, which ranks by version and never by date. Deleting one drops every Elixir image built on it out of the expected set, and erlang/0 will not put it back because its own recency filter passed the tag long ago — a new Elixir release would then silently never build for a base image that is still current. They are 4% of the rows.

Deploy

Order matters in step 2: Application.start/2 validates every module in BOB_LOCAL_JOBS with Code.ensure_loaded? and matches the result against true, so pointing the current image at Bob.Job.DockerCleanup before it exists crashes both replicas on boot.

  • Merge this PR. CI pushes gcr.io/hexpm-prod/bob:<sha>.
  • On hexpm-ops bob-docker-cleanup-job, add the newTag bump in kubernetes/overlays/prod/bob/kustomization.yaml (currently 550e071) alongside the BOB_LOCAL_JOBS change already on the branch. Both must land in the same apply.
  • Merge the ops PR and apply:
    kubectl kustomize --load-restrictor LoadRestrictionsNone kubernetes/overlays/prod/ | kubectl apply -f - --prune -l management=declarative
  • Watch the rollout. The migration builds a CREATE INDEX CONCURRENTLY over 3.08M rows at pod start, before the app boots, so pods take longer than usual to become ready.
  • Confirm the index came out valid — an interrupted build leaves it INVALID, the planner ignores it, and IF NOT EXISTS would skip the corpse on any retry:
    SELECT indisvalid FROM pg_index WHERE indexrelid = 'docker_tags_repo_built_at_index'::regclass;
  • Confirm the job is actually claimed: Bob.Job.DockerCleanup should appear in the jobs dashboard after 02:00, not sit queued.
  • Read the dry run. It logs the backlog and what one scheduled run would take, and should report ~1.97M across the two elixir per-arch repos. To trigger it early:
    ./bin/kubeexec bob --prod -- bin/bob rpc 'Bob.DockerCleanup.run(mode: :dry_run)'
  • Spot-check the premise on one tag before deleting 1.97M: delete a single old per-arch tag by hand, then confirm docker pull hexpm/elixir:<same tag> still resolves. Everything here rests on the multi-arch manifest holding its own layers rather than referencing the per-arch repo.
  • Start the drain, detached — a bare rpc dies with the shell, and the nightly job is killed at three hours by the runner so it cannot do this itself:
    ./bin/kubeexec bob --prod -- bin/bob rpc 'Task.Supervisor.start_child(Bob.Tasks, fn -> Bob.DockerCleanup.drain() end)'
    This deletes regardless of BOB_DOCKER_CLEANUP_MODE, so no config change and no restart is needed to run it.
  • Watch the first hour. The floor is Docker Hub's rate limit: ~1.97M deletes at 600 req/min is roughly 55 hours. The limiter self-calibrates from x-ratelimit-* response headers, so read the real throughput off the logs rather than trusting that estimate. Progress is committed per chunk of 100, so a pod restart mid-drain resumes rather than starting over.
  • Once drained, enable the nightly job for steady state: add BOB_DOCKER_CLEANUP_MODE = "live" to bob-config in terraform/prod/bob.tf, apply, then kubectl rollout restart deployment/bob-deployment. The ConfigMap is consumed with envFrom, so running pods do not pick up a changed value. Steady state is only tens of tags a night, well inside the 10,000 default batch.
  • Reclaim the disk: pg_repack (or VACUUM FULL, which locks) plus REINDEX CONCURRENTLY. DELETE only marks space reusable — docker_tags_repo_tag_index alone is 506MB. The query speedup does not wait for this; the plan flips once autovacuum refreshes the stats.

Rolling back the mode is a terraform change plus a restart. Rolling back the deletions is not possible, which is what the dry run and the single-tag spot-check are for.

ericmj added 6 commits July 29, 2026 15:28
Track each tag's last-pulled time and use it, with build age, to retire
tags Bob no longer needs to keep, while letting a build request pin an
image permanently.

- Parse Docker Hub's `tag_last_pulled` and thread it through reconcile,
  staging and the swap into a new `docker_tags.last_pulled` column.
- Add `Bob.Job.DockerCleanup`, scheduled daily and shipping in `:dry_run`
  (logs per-repo counts, deletes nothing). Live mode deletes, rate-limited
  and in bounded batches, removing a row only once Docker Hub confirms:
  per-arch tags built over 30 days ago, and manifest tags not pulled in
  180 days (falling back to build age when a tag has no recorded pull).
- A pending or completed build request reserves its tags permanently and
  exempts them from cleanup. Requesting an already-built image records the
  reservation without rebuilding; `prune` now keeps completed reservations
  and only reclaims expired dead-ends. Reservation is matched by a tag-name
  hash anti-join against the small build_requests table, not per-row JSONB.
- Document retention in the README and on the request page, and flag
  reserved tags on the Docker tags page.
Give users a way to see whether a tag is due for cleanup.

- Add a "last pulled" column alongside "built".
- Flag each tag's retention outlook with a badge: green "reserved" for
  tags pinned by a build request, or amber "removing"/"removing ~Nd" once
  a tag is within 14 days of (or past) its removal date.
- The outlook comes from `Bob.DockerCleanup.removal_at/3`, the same rule
  the cleanup deletes by, so the page and the deleter never disagree:
  per-arch tags by build age, manifest tags by last-pulled (falling back
  to build age). Reserved tags are never flagged as removing.
- Report every Docker Hub request outcome to the rate limiter via a shared
  paced request, reclaiming the calibration probe's slot on failure so a
  failed delete in the long-lived cleanup job can't wedge the gate for all
  Docker Hub traffic; the pager and delete_tag now share one throttle protocol
- Preserve last_pulled when a reconcile page omits it, so a transient null
  from Docker Hub can't expose a pulled tag to cleanup
- Treat a manifest tag as stale only when neither pulled nor built since the
  cutoff, mirrored in removal_at, so a re-pushed tag isn't deleted and
  immediately rebuilt with a 404 window in between
- Re-check reservations immediately before each delete: the rate-limited
  batch runs for hours and a request made mid-run must still pin its tag
- Complete fully built requests before os_version rotation or TTL expiry so
  a reservation backed by a pending row can't lapse after the image exists
- Satisfy requests from a complete manifest so an image whose per-arch
  staging tags were cleaned is reserved for free instead of rebuilt
- Budget the cleanup batch limit across both repo groups instead of per query
- Serialize reserve() on the target so concurrent submits of the same
  already-built image can't insert duplicate permanent rows
- Assert the SQL reservation predicate matches request_target/1 so the two
  tag-format encodings can't drift silently
A live run now only deletes from the repo groups named in
:docker_cleanup_scope, defaulting to [:per_arch]. The batch limit budgeted a
run but did not confine it: once the per-arch backlog dropped below the limit,
the remainder spilled into the manifest repos on its own, so the first run to
touch the images users pull would have been the one nobody was watching.

Per-arch cleanup keys on built_at, which every row already carries, while
manifest cleanup keys on last_pulled, which is null until Reconcile has swapped
the repo. Separating the groups lets per-arch run while last_pulled is still
backfilling. Dry runs keep reporting both groups whatever the scope, since the
out-of-scope counts are what say when widening is safe.

Both the mode and the scope now read from the environment in prod, so widening
or reverting is a config change rather than a redeploy.

Also pins the retention windows against each other. The checker's expected set
is filtered on release recency and a tag is always built after the release that
triggered it, so per_arch_max_age_days must stay >= build_freshness_days or
each night's cleanup hands the next morning's checker a pile of rebuilds. The
two constants were already equal by luck; a test asserts it now.
The rule was going to delete manifest tags neither pulled nor built in 180
days, reading Docker Hub's tag_last_pulled to decide. That field does not mean
what the rule needed it to mean.

Measured against live Docker Hub on 2026-07-29: 99.5% of all 36,117
hexpm/erlang tags share a single last_pulled date, 2026-07-25, and no tag in
the repo is more than five days idle. hexpm/elixir shows the same pattern
spread over five days, which is what the same crawl looks like against 27x the
tags. Something walks every tag in both repos every ~80 days and resets the
clock; watching three 2019-2020 tags for fifteen minutes caught two of them
being touched 23 seconds apart.

So the threshold sits above the sweep interval and the rule can never fire on
disuse. The 0.33% of hexpm/elixir that does cross 180 days is whichever tags
the last sweep happened to miss — about 3,200 rows out of 3.08 million, chosen
close to arbitrarily, with the blast radius set by someone else's crawl
schedule.

Manifest repos are now simply not pruned. That leaves per-arch cleanup as the
only rule, which is where all the measured benefit was anyway: it removes
2.04M of the table's 3.08M rows and takes the manifest-diff query from ~580ms
to ~9ms. With one rule left the scope flag has nothing to select between, so
it goes too, along with the column, its migration, its index, and the
"last pulled" column on the tags page — a sweep timestamp presented as usage
would mislead anyone reading it.

The README said the images users pull are removed after 180 days. They are
not, and now it says so.
A scheduled job that backs off stops re-running on its interval: one transient
Docker Hub failure would suppress the nightly cleanup for 30 minutes, doubling
to a day. Every other queued job on the master schedule was already exempt.

The new test walks the schedule rather than naming modules, so the next job
added there fails the suite instead of silently backing off in production.
@ericmj
ericmj force-pushed the docker-tag-cleanup branch from 2f1091c to d5b92ec Compare July 29, 2026 22:29
ericmj added 11 commits July 29, 2026 21:14
Stop pruning the erlang per-arch repos. DockerChecker.expected_elixir_tags/0
derives the entire Elixir build matrix from the erlang per-arch rows in the
mirror, filtered by latest_erlang_versions/0, which ranks by version and never
by date. Deleting an erlang per-arch tag drops every Elixir image built on it
out of the expected set, and erlang/0 will not put it back because its own
recency filter passed the tag long ago. A new Elixir release would then
silently never build for a base image that is still current. The earlier
"cleanup can only reach tags the checker has stopped expecting" argument holds
for the erlang pass only; the Elixir pass reads the mirror, not release dates.
The elixir per-arch repos carry no such dependency and are 96% of the rows.

Commit deletions in chunks instead of once at the end. The batch was 100,000
serial deletes paced at 600 req/min, so ~2.8h against Runner's 3h job timeout,
which hard-kills the task. Every run died before recording anything: tags gone
from Docker Hub, every row still present, the same candidates re-selected the
next night, 404s all the way, no progress ever. The batch default drops to
10,000 so a run finishes well inside the timeout, and BOB_DOCKER_CLEANUP_BATCH
can lower it without a deploy.

Re-check built_at alongside the reservation before each delete. The nightly
reconcile can bump a tag's built_at from a re-push while a long batch is still
working through candidates selected before it.

Abort after 25 consecutive failures. A token without delete permission 401s on
every candidate, which was otherwise hours of full-rate Docker Hub traffic and
one error line per tag, every night.

Report what a live run would actually delete in the dry run. The count was
unbounded while the run is capped at the batch, so the number read before
flipping to :live overstated one night by orders of magnitude.

Drop the index before creating it in the cleanup migration. An interrupted
CREATE INDEX CONCURRENTLY leaves an INVALID index that the planner ignores and
that IF NOT EXISTS then skips by name, so a retry reports success while every
cleanup query seq-scans the table.

Fix two tests that could not fail. The backoff guard walked :master_schedule,
which config/test.exs sets to [], so it asserted nothing and the exemption had
no coverage; it now lists the modules. The removing-badge helper anchored its
match directly after </code>, so the reserved span pushed it out of the pattern
and the "reserved tags are not also flagged removing" refute was unfalsifiable.

Correct the retention copy on the request page, which still advertised the
dropped 180-day rule, and in the README, which still sold reservation as
protecting images that are no longer deletable. Revert the tags table widths
raised for the removed column, and correct the moduledoc, which overstated the
crawl measurements it cites.
Three ordering assertions used >= and <= directly on DateTime structs. Term
order on a struct is map key order, which puts :microsecond ahead of :second,
so two timestamps straddling a second boundary compare backwards:
04:16:41.001502 is not >= 04:16:40.998349. They passed only while both
timestamps happened to land in the same second.

Unrelated to the cleanup work and pre-existing on main, but it failed CI on
this branch.
Deletes ran one at a time while the tag reader runs 50 in parallel, so a run
was bound by round-trip latency rather than by Docker Hub's budget and used a
fraction of the rate it was allowed. They now run 25 at a time; the shared rate
limiter is still the only real throttle, so this changes how fast the budget is
spent, not how much.

The per-tag reservation and built_at re-check becomes one query per chunk
instead of one per tag. The prod pool is 10 connections and shared with the web
endpoint, so a per-tag round trip under concurrency would have starved it. The
window in which a mid-run reservation can be missed widens from microseconds to
one chunk, which is still far tighter than the selection-time-only check it
replaced.

drain/1 loops runs until a pass deletes nothing, for the ~1.97M tag initial
backlog that no three-hour job can finish. It is not the scheduled path and is
meant to be run detached, since the nightly job is killed at three hours by the
runner. Progress is committed per chunk, so a restart mid-drain resumes rather
than starting over, and a pass that deletes nothing ends the loop so a tag
Docker Hub keeps rejecting cannot spin it forever.

The error ceiling moves from consecutive failed tags to consecutive chunks with
no successful delete, since "consecutive" is meaningless once deletes overlap.
It is an explicit clear-the-backlog command and looping a dry run would spin,
so it overrides :docker_cleanup_mode. Worth saying out loud, since the mode is
otherwise the safety catch.
Added when the batch size was the only lever against the three-hour job
timeout. drain/1 handles the initial backlog now and takes :limit directly, and
steady state is tens of tags a night, so there is no case left where anyone
tunes this from the environment.
The cleanup matched reservations with a raw SQL predicate that rebuilt the tag
name from build_requests columns with a CASE, interpolated into four queries.
That forced every query touching it to be raw, and it duplicated
request_target/1 in SQL, so the two encodings needed a test to keep them in
step. Drift in either would have silently stopped reservations matching and let
the cleanup delete tags users had pinned.

BuildRequest.changeset/2 now derives the target once and stores it, so the rule
lives in one place and matching is an equality on an indexed column. All six
cleanup queries become Ecto, including delete_docker_tags, which never needed
the predicate and was raw only by association. Net raw query count in
Bob.Artifacts is back to what it was on main.

Postgres plans it the same way, measured on the prod-scale snapshot: a hash
anti-join with build_requests on the hash side, 337ms over the full 2.05M
backlog against the 184ms the CASE version took for the same shape, the
difference being that this ran without the (repo, built_at) index the migration
adds. No CASE evaluated per scanned row now.
They had turned into PR-description prose: measurements with dates and
percentages, the reasoning behind rejected designs, and an rpc invocation
copied into a docstring. None of that describes the code, and it goes stale the
moment anything moves. The reasoning lives in the PR and the commit history.

What is left says why a constant has the value it does, why a query is shaped
the way it is, and which invariant a test is pinning.
The tags page no longer flags tags as removing. That took removal_at/2 with it,
which existed only to feed the badge, along with its styles and tests.

The target column is gone too. It was added to stop the reservation match from
rebuilding the tag name in SQL, but build_requests is small enough to just
collect the reserved tag names and match against the list, which needs no
column, no migration and no backfill. BuildRequest.target/1 stays as the one
place the tag name is derived.
docker_cleanup_per_arch_repos/0 had no callers once removal_at/2 went.
docker_tag_reserved?/2 had none either, deletable_docker_tags/2 does that work
now. The dry run computed two map keys nothing read.
backoff?/1 is private again; the test drives the same behaviour through
Queue.add/start/failure, which is what production does anyway and still fails
when a scheduled job loses its exemption.

per_arch_max_age_days/0 and build_freshness_days/0 existed solely so one test
could compare the two constants. Both constants already carry a comment naming
the other, so the accessors and the test are gone.
run/1 took :chunk and :concurrency purely so tests could force a chunk per tag
and serial deletes. Two of those tests then asserted a guarantee production
does not give: at the real concurrency a whole chunk is in flight before a
reservation made mid-run can land, because the re-check happens per chunk, not
per tag.

The tests now insert more than one chunk of tags and assert what the code
actually promises: a reservation or a re-push during one chunk spares the next
chunk, not the current one. Both options are gone.

Rows are bulk-inserted since these need a few hundred; add_docker_tag/4 that
many times dominated the runtime.
@ericmj
ericmj marked this pull request as ready for review July 30, 2026 17:47
@ericmj
ericmj merged commit 1e215ab into main Jul 30, 2026
4 checks passed
@ericmj
ericmj deleted the docker-tag-cleanup branch July 30, 2026 17:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant