Repository navigation
Docker tag cleanup - #274
Merged
Merged
Conversation
Track each tag's last-pulled time and use it, with build age, to retire tags Bob no longer needs to keep, while letting a build request pin an image permanently. - Parse Docker Hub's `tag_last_pulled` and thread it through reconcile, staging and the swap into a new `docker_tags.last_pulled` column. - Add `Bob.Job.DockerCleanup`, scheduled daily and shipping in `:dry_run` (logs per-repo counts, deletes nothing). Live mode deletes, rate-limited and in bounded batches, removing a row only once Docker Hub confirms: per-arch tags built over 30 days ago, and manifest tags not pulled in 180 days (falling back to build age when a tag has no recorded pull). - A pending or completed build request reserves its tags permanently and exempts them from cleanup. Requesting an already-built image records the reservation without rebuilding; `prune` now keeps completed reservations and only reclaims expired dead-ends. Reservation is matched by a tag-name hash anti-join against the small build_requests table, not per-row JSONB. - Document retention in the README and on the request page, and flag reserved tags on the Docker tags page.
Give users a way to see whether a tag is due for cleanup. - Add a "last pulled" column alongside "built". - Flag each tag's retention outlook with a badge: green "reserved" for tags pinned by a build request, or amber "removing"/"removing ~Nd" once a tag is within 14 days of (or past) its removal date. - The outlook comes from `Bob.DockerCleanup.removal_at/3`, the same rule the cleanup deletes by, so the page and the deleter never disagree: per-arch tags by build age, manifest tags by last-pulled (falling back to build age). Reserved tags are never flagged as removing.
- Report every Docker Hub request outcome to the rate limiter via a shared paced request, reclaiming the calibration probe's slot on failure so a failed delete in the long-lived cleanup job can't wedge the gate for all Docker Hub traffic; the pager and delete_tag now share one throttle protocol - Preserve last_pulled when a reconcile page omits it, so a transient null from Docker Hub can't expose a pulled tag to cleanup - Treat a manifest tag as stale only when neither pulled nor built since the cutoff, mirrored in removal_at, so a re-pushed tag isn't deleted and immediately rebuilt with a 404 window in between - Re-check reservations immediately before each delete: the rate-limited batch runs for hours and a request made mid-run must still pin its tag - Complete fully built requests before os_version rotation or TTL expiry so a reservation backed by a pending row can't lapse after the image exists - Satisfy requests from a complete manifest so an image whose per-arch staging tags were cleaned is reserved for free instead of rebuilt - Budget the cleanup batch limit across both repo groups instead of per query - Serialize reserve() on the target so concurrent submits of the same already-built image can't insert duplicate permanent rows - Assert the SQL reservation predicate matches request_target/1 so the two tag-format encodings can't drift silently
A live run now only deletes from the repo groups named in :docker_cleanup_scope, defaulting to [:per_arch]. The batch limit budgeted a run but did not confine it: once the per-arch backlog dropped below the limit, the remainder spilled into the manifest repos on its own, so the first run to touch the images users pull would have been the one nobody was watching. Per-arch cleanup keys on built_at, which every row already carries, while manifest cleanup keys on last_pulled, which is null until Reconcile has swapped the repo. Separating the groups lets per-arch run while last_pulled is still backfilling. Dry runs keep reporting both groups whatever the scope, since the out-of-scope counts are what say when widening is safe. Both the mode and the scope now read from the environment in prod, so widening or reverting is a config change rather than a redeploy. Also pins the retention windows against each other. The checker's expected set is filtered on release recency and a tag is always built after the release that triggered it, so per_arch_max_age_days must stay >= build_freshness_days or each night's cleanup hands the next morning's checker a pile of rebuilds. The two constants were already equal by luck; a test asserts it now.
The rule was going to delete manifest tags neither pulled nor built in 180 days, reading Docker Hub's tag_last_pulled to decide. That field does not mean what the rule needed it to mean. Measured against live Docker Hub on 2026-07-29: 99.5% of all 36,117 hexpm/erlang tags share a single last_pulled date, 2026-07-25, and no tag in the repo is more than five days idle. hexpm/elixir shows the same pattern spread over five days, which is what the same crawl looks like against 27x the tags. Something walks every tag in both repos every ~80 days and resets the clock; watching three 2019-2020 tags for fifteen minutes caught two of them being touched 23 seconds apart. So the threshold sits above the sweep interval and the rule can never fire on disuse. The 0.33% of hexpm/elixir that does cross 180 days is whichever tags the last sweep happened to miss — about 3,200 rows out of 3.08 million, chosen close to arbitrarily, with the blast radius set by someone else's crawl schedule. Manifest repos are now simply not pruned. That leaves per-arch cleanup as the only rule, which is where all the measured benefit was anyway: it removes 2.04M of the table's 3.08M rows and takes the manifest-diff query from ~580ms to ~9ms. With one rule left the scope flag has nothing to select between, so it goes too, along with the column, its migration, its index, and the "last pulled" column on the tags page — a sweep timestamp presented as usage would mislead anyone reading it. The README said the images users pull are removed after 180 days. They are not, and now it says so.
A scheduled job that backs off stops re-running on its interval: one transient Docker Hub failure would suppress the nightly cleanup for 30 minutes, doubling to a day. Every other queued job on the master schedule was already exempt. The new test walks the schedule rather than naming modules, so the next job added there fails the suite instead of silently backing off in production.
ericmj
force-pushed
the
docker-tag-cleanup
branch
from
July 29, 2026 22:29
2f1091c to
d5b92ec
Compare
Stop pruning the erlang per-arch repos. DockerChecker.expected_elixir_tags/0 derives the entire Elixir build matrix from the erlang per-arch rows in the mirror, filtered by latest_erlang_versions/0, which ranks by version and never by date. Deleting an erlang per-arch tag drops every Elixir image built on it out of the expected set, and erlang/0 will not put it back because its own recency filter passed the tag long ago. A new Elixir release would then silently never build for a base image that is still current. The earlier "cleanup can only reach tags the checker has stopped expecting" argument holds for the erlang pass only; the Elixir pass reads the mirror, not release dates. The elixir per-arch repos carry no such dependency and are 96% of the rows. Commit deletions in chunks instead of once at the end. The batch was 100,000 serial deletes paced at 600 req/min, so ~2.8h against Runner's 3h job timeout, which hard-kills the task. Every run died before recording anything: tags gone from Docker Hub, every row still present, the same candidates re-selected the next night, 404s all the way, no progress ever. The batch default drops to 10,000 so a run finishes well inside the timeout, and BOB_DOCKER_CLEANUP_BATCH can lower it without a deploy. Re-check built_at alongside the reservation before each delete. The nightly reconcile can bump a tag's built_at from a re-push while a long batch is still working through candidates selected before it. Abort after 25 consecutive failures. A token without delete permission 401s on every candidate, which was otherwise hours of full-rate Docker Hub traffic and one error line per tag, every night. Report what a live run would actually delete in the dry run. The count was unbounded while the run is capped at the batch, so the number read before flipping to :live overstated one night by orders of magnitude. Drop the index before creating it in the cleanup migration. An interrupted CREATE INDEX CONCURRENTLY leaves an INVALID index that the planner ignores and that IF NOT EXISTS then skips by name, so a retry reports success while every cleanup query seq-scans the table. Fix two tests that could not fail. The backoff guard walked :master_schedule, which config/test.exs sets to [], so it asserted nothing and the exemption had no coverage; it now lists the modules. The removing-badge helper anchored its match directly after </code>, so the reserved span pushed it out of the pattern and the "reserved tags are not also flagged removing" refute was unfalsifiable. Correct the retention copy on the request page, which still advertised the dropped 180-day rule, and in the README, which still sold reservation as protecting images that are no longer deletable. Revert the tags table widths raised for the removed column, and correct the moduledoc, which overstated the crawl measurements it cites.
Three ordering assertions used >= and <= directly on DateTime structs. Term order on a struct is map key order, which puts :microsecond ahead of :second, so two timestamps straddling a second boundary compare backwards: 04:16:41.001502 is not >= 04:16:40.998349. They passed only while both timestamps happened to land in the same second. Unrelated to the cleanup work and pre-existing on main, but it failed CI on this branch.
Deletes ran one at a time while the tag reader runs 50 in parallel, so a run was bound by round-trip latency rather than by Docker Hub's budget and used a fraction of the rate it was allowed. They now run 25 at a time; the shared rate limiter is still the only real throttle, so this changes how fast the budget is spent, not how much. The per-tag reservation and built_at re-check becomes one query per chunk instead of one per tag. The prod pool is 10 connections and shared with the web endpoint, so a per-tag round trip under concurrency would have starved it. The window in which a mid-run reservation can be missed widens from microseconds to one chunk, which is still far tighter than the selection-time-only check it replaced. drain/1 loops runs until a pass deletes nothing, for the ~1.97M tag initial backlog that no three-hour job can finish. It is not the scheduled path and is meant to be run detached, since the nightly job is killed at three hours by the runner. Progress is committed per chunk, so a restart mid-drain resumes rather than starting over, and a pass that deletes nothing ends the loop so a tag Docker Hub keeps rejecting cannot spin it forever. The error ceiling moves from consecutive failed tags to consecutive chunks with no successful delete, since "consecutive" is meaningless once deletes overlap.
It is an explicit clear-the-backlog command and looping a dry run would spin, so it overrides :docker_cleanup_mode. Worth saying out loud, since the mode is otherwise the safety catch.
Added when the batch size was the only lever against the three-hour job timeout. drain/1 handles the initial backlog now and takes :limit directly, and steady state is tens of tags a night, so there is no case left where anyone tunes this from the environment.
The cleanup matched reservations with a raw SQL predicate that rebuilt the tag name from build_requests columns with a CASE, interpolated into four queries. That forced every query touching it to be raw, and it duplicated request_target/1 in SQL, so the two encodings needed a test to keep them in step. Drift in either would have silently stopped reservations matching and let the cleanup delete tags users had pinned. BuildRequest.changeset/2 now derives the target once and stores it, so the rule lives in one place and matching is an equality on an indexed column. All six cleanup queries become Ecto, including delete_docker_tags, which never needed the predicate and was raw only by association. Net raw query count in Bob.Artifacts is back to what it was on main. Postgres plans it the same way, measured on the prod-scale snapshot: a hash anti-join with build_requests on the hash side, 337ms over the full 2.05M backlog against the 184ms the CASE version took for the same shape, the difference being that this ran without the (repo, built_at) index the migration adds. No CASE evaluated per scanned row now.
They had turned into PR-description prose: measurements with dates and percentages, the reasoning behind rejected designs, and an rpc invocation copied into a docstring. None of that describes the code, and it goes stale the moment anything moves. The reasoning lives in the PR and the commit history. What is left says why a constant has the value it does, why a query is shaped the way it is, and which invariant a test is pinning.
The tags page no longer flags tags as removing. That took removal_at/2 with it, which existed only to feed the badge, along with its styles and tests. The target column is gone too. It was added to stop the reservation match from rebuilding the tag name in SQL, but build_requests is small enough to just collect the reserved tag names and match against the list, which needs no column, no migration and no backfill. BuildRequest.target/1 stays as the one place the tag name is derived.
docker_cleanup_per_arch_repos/0 had no callers once removal_at/2 went. docker_tag_reserved?/2 had none either, deletable_docker_tags/2 does that work now. The dry run computed two map keys nothing read.
backoff?/1 is private again; the test drives the same behaviour through Queue.add/start/failure, which is what production does anyway and still fails when a scheduled job loses its exemption. per_arch_max_age_days/0 and build_freshness_days/0 existed solely so one test could compare the two constants. Both constants already carry a comment naming the other, so the accessors and the test are gone.
run/1 took :chunk and :concurrency purely so tests could force a chunk per tag and serial deletes. Two of those tests then asserted a guarantee production does not give: at the real concurrency a whole chunk is in flight before a reservation made mid-run can land, because the re-check happens per chunk, not per tag. The tests now insert more than one chunk of tags and assert what the code actually promises: a reservation or a re-push during one chunk spares the next chunk, not the current one. Both options are gone. Rows are bulk-inserted since these need a few hundred; add_docker_tag/4 that many times dominated the runtime.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prunes
hexpm/elixir-amd64andhexpm/elixir-arm64, which hold the single-arch images the multi-arch manifests are assembled from rather than images anyone pulls directly. A tag built more than 30 days ago is removed and the multi-arch image it backs stays available. Requesting an image now reserves it from cleanup permanently, and reserves it without rebuilding when it already exists.That is ~1.97M of the table's 3.08M rows. It also fixes
manifest_mismatches/3, the diff the checker runs every 15 minutes for both erlang and elixir: measured against a prod-scale snapshot it drops from ~580ms warm (~3.4s cold, ~1.6GB of buffer reads) to ~9ms, because the plan stops hash-joining a million rows per side and becomes a nested loop over a few thousand.What it deliberately does not do
The manifest repos are never pruned. The rule was going to delete
hexpm/erlangandhexpm/elixirtags neither pulled nor built in 180 days, reading Docker Hub'stag_last_pulled. That field does not mean what the rule needed. Measured 2026-07-29: 99.5% of all 36,117hexpm/erlangtags share the single pull date 2026-07-25 and nothing in the repo is more than five days idle;hexpm/elixirshows the same pattern spread over five days. Something walks every tag in both repos roughly every 80 days and resets the field. The threshold sat above the crawl interval, so the rule could never fire on disuse — the 0.33% that crossed 180 days were the tags the last crawl happened to miss, about 3,200 rows out of 3.08M, chosen close to arbitrarily and with the blast radius set by someone else's schedule. Thelast_pulledcolumn, its migration and its index went with it.The erlang per-arch repos are excluded. They look equally disposable and are not.
DockerChecker.expected_elixir_tags/0derives the entire Elixir build matrix from the erlang per-arch rows in the mirror, filtered bylatest_erlang_versions/0, which ranks by version and never by date. Deleting one drops every Elixir image built on it out of the expected set, anderlang/0will not put it back because its own recency filter passed the tag long ago — a new Elixir release would then silently never build for a base image that is still current. They are 4% of the rows.Deploy
Order matters in step 2:
Application.start/2validates every module inBOB_LOCAL_JOBSwithCode.ensure_loaded?and matches the result againsttrue, so pointing the current image atBob.Job.DockerCleanupbefore it exists crashes both replicas on boot.gcr.io/hexpm-prod/bob:<sha>.bob-docker-cleanup-job, add thenewTagbump inkubernetes/overlays/prod/bob/kustomization.yaml(currently550e071) alongside theBOB_LOCAL_JOBSchange already on the branch. Both must land in the same apply.kubectl kustomize --load-restrictor LoadRestrictionsNone kubernetes/overlays/prod/ | kubectl apply -f - --prune -l management=declarativeCREATE INDEX CONCURRENTLYover 3.08M rows at pod start, before the app boots, so pods take longer than usual to become ready.INVALID, the planner ignores it, andIF NOT EXISTSwould skip the corpse on any retry:SELECT indisvalid FROM pg_index WHERE indexrelid = 'docker_tags_repo_built_at_index'::regclass;Bob.Job.DockerCleanupshould appear in the jobs dashboard after 02:00, not sit queued../bin/kubeexec bob --prod -- bin/bob rpc 'Bob.DockerCleanup.run(mode: :dry_run)'docker pull hexpm/elixir:<same tag>still resolves. Everything here rests on the multi-arch manifest holding its own layers rather than referencing the per-arch repo../bin/kubeexec bob --prod -- bin/bob rpc 'Task.Supervisor.start_child(Bob.Tasks, fn -> Bob.DockerCleanup.drain() end)'This deletes regardless of
BOB_DOCKER_CLEANUP_MODE, so no config change and no restart is needed to run it.x-ratelimit-*response headers, so read the real throughput off the logs rather than trusting that estimate. Progress is committed per chunk of 100, so a pod restart mid-drain resumes rather than starting over.BOB_DOCKER_CLEANUP_MODE = "live"tobob-configinterraform/prod/bob.tf, apply, thenkubectl rollout restart deployment/bob-deployment. The ConfigMap is consumed withenvFrom, so running pods do not pick up a changed value. Steady state is only tens of tags a night, well inside the 10,000 default batch.pg_repack(orVACUUM FULL, which locks) plusREINDEX CONCURRENTLY.DELETEonly marks space reusable —docker_tags_repo_tag_indexalone is 506MB. The query speedup does not wait for this; the plan flips once autovacuum refreshes the stats.Rolling back the mode is a terraform change plus a restart. Rolling back the deletions is not possible, which is what the dry run and the single-tag spot-check are for.