Skip to content

Commit defec8e

Browse files
committed
Diff Docker manifest archs with a full join instead of an aggregate
manifest_mismatches/3 aggregated both per-arch repos' full tag histories (GROUP BY tag + array_agg over ~1.9M rows for elixir), forcing a sort of the whole set on every run. With work_mem below the sort's ~218MB footprint that spills to a fast external merge, but past it the planner switches to a single in-memory quicksort doing full locale-collated comparisons, which is ~4x slower — the regime production runs in, at ~20s per elixir call every 15 minutes. Tags are unique per (repo, tag), so the built-arch set is simply which of the two per-arch repos contain the tag. A full join of two index-only scans computes the same result with no sort or aggregate and also parallelizes: 3.4s to 0.57s for elixir and 264ms to 9.7ms for erlang on a production-sized snapshot, flat across work_mem settings, with the generic prepared-statement plan holding at ~0.6s. Verified against the old query on synthesized mismatches of every class (missing manifest, manifest lacking a built arch, single-arch tag with and without coverage, manifest-only tag): identical rows in both directions.
1 parent d72db9f commit defec8e

1 file changed

Lines changed: 12 additions & 7 deletions

File tree

‎lib/bob/artifacts.ex‎

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -431,18 +431,23 @@ defmodule Bob.Artifacts do
431431
as is a manifest carrying more archs than were built.
432432
"""
433433
def manifest_mismatches(manifest_repo, amd64_repo, arm64_repo) do
434-
# The ::text[] cast is required: archs is varchar[] and array containment
435-
# does not unify varchar[] with the aggregated text[].
434+
# Tags are unique per repo, so the built-arch sets come from a full join of
435+
# two index-only scans on (repo, tag) — no sort or aggregate over the
436+
# repos' full histories, which also keeps the plan fast regardless of
437+
# work_mem (a GROUP BY formulation degrades ~4x once its sort fits in
438+
# memory). The ::text[] cast is required: archs is varchar[] and array
439+
# containment does not unify varchar[] with text[].
436440
%{rows: rows} =
437441
Repo.query!(
438442
"""
439443
SELECT p.tag, p.archs
440444
FROM (
441-
SELECT tag,
442-
array_agg(DISTINCT CASE WHEN repo = $2 THEN 'amd64' ELSE 'arm64' END) AS archs
443-
FROM docker_tags
444-
WHERE repo IN ($2, $3)
445-
GROUP BY tag
445+
SELECT COALESCE(a.tag, b.tag) AS tag,
446+
CASE WHEN b.tag IS NULL THEN ARRAY['amd64']
447+
WHEN a.tag IS NULL THEN ARRAY['arm64']
448+
ELSE ARRAY['amd64', 'arm64'] END AS archs
449+
FROM (SELECT tag FROM docker_tags WHERE repo = $2) a
450+
FULL JOIN (SELECT tag FROM docker_tags WHERE repo = $3) b ON a.tag = b.tag
446451
) AS p
447452
LEFT JOIN docker_tags m ON m.repo = $1 AND m.tag = p.tag
448453
WHERE m.id IS NULL OR NOT (m.archs::text[] @> p.archs)

0 commit comments

Comments
 (0)