Skip to content

Give the sdist check a build directory its cache can key on - #152

Merged
martin-s-a merged 1 commit into
mainfrom
cache-the-sdist-build
Sep 19, 2026
Merged

martin-s-a merged 1 commit into
mainfrom
cache-the-sdist-build

Conversation

@martin-s-a

Copy link
Copy Markdown
Contributor

Verify the source distribution hits 19 of 334 compilations. The same
figure every run, never converging, with the cache restoring correctly and
every call cacheable.

Why

#136 diagnosed it
and left it: scikit-build-core builds in a unique temporary directory that has
no fixed relation to the one pip unpacked the archive into. Two independent
random path components, so the sources sit somewhere different relative to the
build directory every run and CCACHE_BASEDIR has nothing stable to rewrite
against. The 19 hits are the dependencies that happen to live inside the build
directory — 12 of Boost and 7 of spdlog.

What #136 did not reach for is scikit-build-core's own build-dir, whose
default is documented as "a unique temporary directory" and which the docs say
gives "major rebuild speedups" when made persistent.

Why relative, specifically

The value has to be relative, and that is the whole of the fix. Measured
against scikit-build-core 1.0.3, the version this project pins:

(default)                 CMAKE_BINARY_DIR = /tmp/tmpzolfmuz5/build
SKBUILD_BUILD_DIR=build   CMAKE_BINARY_DIR = <source>/build

It is resolved against the source directory, which puts the working directory
inside the source tree at a fixed depth. Every absolute path then rewrites to
the same relative string from one run to the next — the sources to ../…, the
fetched dependencies to ../../fetched/… — because the random element is a
single directory name at a known position. An absolute build directory
would fix nothing: the sources would still be at a random path.

Cost and effect

The job takes 17m06s, the second-longest in the deploy run, about two minutes
short of the critical path the Windows AMD64 wheels hold at 18m58s. So this
buys billed minutes today rather than wall clock — the reason to do it now is
the headroom before it becomes the job everything waits on.

It arrives as an environment variable on the step, which is the mechanism
#134 already chose
for this job's cache settings and its CMake defines, so pip install dist/*.tar.gz stays the command a user would run. Nothing about what is
compiled, with which flags, or into what layout changes — only where the
intermediates are written — so it cannot mask the packaging defect the job
exists to catch.

Measuring it

Not on this run. New paths mean new keys, so the run that adopts this misses
everything and populates; the second deploy after merging is the one whose
hit rate means anything. Same pattern as the coverage flag in rexlib#460.

🤖 Generated with Claude Code

scikit-build-core builds in a temporary directory unrelated to the one pip
unpacks the archive into, so the sources sit somewhere different relative to
the build directory on every run and CCACHE_BASEDIR has nothing stable to
rewrite against. The job has been hitting 19 of 334 compilations, the same
figure every time, and those 19 are the dependencies that happen to live
inside the build directory.

A relative build directory is resolved against the source, which puts the
working directory inside the source tree at a fixed depth and leaves every
rewritten path the same from one run to the next.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@martin-s-a martin-s-a self-assigned this Sep 19, 2026
@martin-s-a martin-s-a added the enhancement New feature or request label Sep 19, 2026
@martin-s-a
martin-s-a merged commit f01083c into main Sep 19, 2026
47 checks passed
@martin-s-a
martin-s-a deleted the cache-the-sdist-build branch September 19, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant