Skip to content

Support reusing unchanged tarballs from an earlier JLL release - #1455

Merged
maleadt merged 2 commits into
masterfrom
tb/partial-reuse
Oct 16, 2026
Merged

maleadt merged 2 commits into
masterfrom
tb/partial-reuse

Conversation

@maleadt

@maleadt maleadt commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Yggdrasil rebuilds every platform of a recipe whenever the recipe changes. For the LLVM *_unified JLLs, which carry all LLVM versions in one package, bumping one LLVM version rebuilt all 968 tarballs; 824 of them came out identical (same tree hash) to the previous release. That was ~35 of the 51 agent-hours of JuliaPackaging/Yggdrasil#15144.

The plan is for Yggdrasil's CI to schedule only the platforms whose inputs changed, and to register the new JLL version with the unchanged platforms pointing at the tarballs of the earlier release. This PR adds what BinaryBuilder needs for that:

  • --meta-json records the build_tarballs keyword arguments that can affect the tarballs (preferred_gcc_version, compilers, skip_audit, ...) as build_options. Without them, two recipe versions with equal sources, script and dependencies could still produce different tarballs. Only explicitly passed options are recorded, and the repr-based values are only meant to be compared under the same Julia and BinaryBuilder versions.
  • rebuild_jll_package accepts reused_urls: tarballs that are not rebuilt and stay where an earlier release published them. Their Artifacts.toml entries keep that URL, and their hashes and product paths come from the build metadata sidecar (read_build_meta(...; verify=false)), so they are neither downloaded nor re-uploaded. A platform that is both rebuilt and reused, or reused twice, is an error.

Registration of Clang_unified currently downloads and re-uploads 39 GB (25 min); with reuse that drops to the ~7 GB that actually changed.

@giordano

Copy link
Copy Markdown
Member

In https://github.com/JuliaPackaging/Yggdrasil/blob/910714b8c451f9d9058b57fe98e99dbc49848bdf/.ci/bb1_project/register_package.jl we already allow reusing binaries from previous releases with [skip build], but my understanding is that this is for a more granular approach were only part of the platforms exists already, is that correct?

@maleadt

maleadt commented Oct 16, 2026

Copy link
Copy Markdown
Member Author

Right. [skip build] is all or nothing, with this JLLs like the unified LLVM ones or the CUDA Runtime typically only rebuild a single version, so they can reuse the others. See the Yggdrasil PR.

Base automatically changed from tb/json1-sidecars to master October 16, 2026 12:44
The meta JSON describes sources, script, products and dependencies, but
not the build_tarballs keyword arguments that also shape the tarballs
(preferred_gcc_version, compilers, skip_audit, ...). Add them as
build_options, so that CI can tell whether a platform's tarball from an
earlier release is still what the recipe would build.
rebuild_jll_package gains reused_urls: tarballs that are not rebuilt but
stay where an earlier release published them. Their Artifacts.toml
entries keep that URL, and their metadata comes from the build metadata
sidecar alone, so they need not be downloaded or re-uploaded. This lets
Yggdrasil skip rebuilding platforms whose inputs did not change, e.g.
the LLVM versions of the *_unified JLLs that a bump does not touch.
@maleadt
maleadt merged commit e1dabe5 into master Oct 16, 2026
3 of 7 checks passed
@maleadt
maleadt deleted the tb/partial-reuse branch October 16, 2026 12:55
maleadt added a commit to JuliaPackaging/Yggdrasil that referenced this pull request Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants