[maleadt]

The *_unified LLVM JLLs carry every supported LLVM version, and CUDA_Runtime and CUDA_Compiler every CUDA toolkit. Bumping one version therefore rebuilds and re-uploads every platform. For the LLVM 23.1.4 bump (#15144) that was 968 jobs in the unified JLLs, and 824 of them produced tarballs with the same tree hash as the previous release: about 35 of the build's 51 agent-hours. Registering Clang_unified then re-uploaded 39 GB, which took 25 minutes.

This PR builds only the platforms whose build changed, and keeps the previous release's tarballs for the rest.

How it works

• .ci/bb1_project/plan_reuse.jl runs from the generator, for an allowlist of projects: the four *_unified LLVM recipes, CUDA_Runtime and CUDA_Compiler. It:
  1. finds the latest registered version of the JLL, and reads from its README the Yggdrasil commit that built it;
  2. evaluates the recipe at that commit with --meta-json, in today's environment;
  3. compares each platform's build description against head: sources (including the git tree of DirectorySources), script, products, build dependencies, and build options.
• Only the platforms that differ are scheduled. The other tarballs are listed in <project>.reuse.json, which the launcher step uploads as an artifact.
• Registration keeps the Artifacts.toml entries of reused platforms verbatim, so they still point at the release that built them. Their product metadata comes from the build metadata sidecars, which every release now publishes alongside its tarballs. For releases made before this change, registration downloads and unpacks each tarball once instead.
• Wrapper-only changes reuse everything: augment_platform_block, init_block, compat bounds, runtime dependencies. Bumping the rebuild trigger rebuilds everything. Any failure while planning falls back to a full build.

Builds on

• Support reusing unchanged tarballs from an earlier JLL release BinaryBuilder.jl#1455 (and Parse build metadata sidecars into Dicts with JSON 1.x BinaryBuilder.jl#1457): rebuild_jll_package(...; reused_urls), and build_options in --meta-json.
• Serialize dependency build versions and git sources to JSON BinaryBuilderBase.jl#503: dependency JSON now includes version pins and git revisions. Without it, a new LLVM_full_jll build is invisible in meta-json, and the planner refuses to reuse anything.

Both are merged; the first commit updates .ci/bb1_project to pick them up.

Validation

Done locally, re-checked after rebasing onto the Julia 1.13.2 CI environment; this PR touches no recipe, so its CI doesn't exercise any of this.

• Planner:
  • Clang_unified with the v23 entry set back to 23.1.3 rebuilds exactly the 36 v23 platforms and reuses 206. Without BinaryBuilderBase.jl#503 it wrongly reuses all 242.
  • An unchanged recipe, or a CUDA_Runtime change confined to toolkit_selection.jl, reuses everything.
  • A product change in build_13.4.jl rebuilds the 3 CUDA 13.4 platforms.
  • A rebuild trigger bump rebuilds all 242.
• Registration, as a dry run of register_package.jl for LLD_unified, simulating 0.1.10 with v23 at 23.1.3. The push, upload and registration were cut off, and buildkite-agent was stubbed.
  • The 206 other entries are copied verbatim from 0.1.9.
  • The 36 v23 entries carry 0.1.8's tree hashes and sha256s under the new release's URLs.
  • The generated JLL loads, selects the reused llvm_version=18 artifact through its 0.1.9 URL, and runs lld.

Open questions

• Builder changes. Both sides of the comparison are evaluated with today's BinaryBuilder, so a change to the builder itself (shards, audit, packaging) doesn't count as a difference. A reused tarball is then as old as one in a JLL whose recipe wasn't touched. Invalidating reuse whenever BinaryBuilder or BinaryBuilderBase changes would void it in most rounds: the manifest changes about weekly, and BinaryBuilderBase's Artifacts.toml changed in 17 of its 44 commits since March. A precise alternative is to record in each sidecar the shards a build used, and compare those per platform. Is that worth doing before enabling this, or is the trigger bump enough as the way to force rebuilds?
• Unpinned dependencies are compared as declared, not as resolved, so a newly registered Zlib_jll doesn't trigger a rebuild. It doesn't matter for these recipes, which only repackage LLVM_full or NVIDIA's archives, but it would for general recipes. Pinned and git-sourced dependencies, like LLVM_full, are compared exactly.

[maleadt]

I'll be rebuilding LLVM today, so I'm going to merge this to test the added functionality. This shouldn't impact other recipes.