Repository navigation
Conversation
Yggdrasil runs the auditor with 16 threads on Julia 1.12 and has seen audits stall indefinitely, something never observed on Julia 1.7. * Make the per-library loop in `check_dynamic_linkage` serial. Every iteration modifies the same binary and serialises on its patchelf lock, so the nested `Threads.@threads` gained nothing. On Julia 1.7 nested `@threads` loops ran serially anyway; since Julia 1.9 they spawn `nthreads()` tasks per outer task, so on CI up to 256 tasks contended on locks and subprocess pipes. This is the only behavioural difference between 1.7 and 1.12 on this code path. * Run the `dlopen` probe with stdout discarded. `open(cmd)` followed by `wait` left an unread stdout pipe open per shared library, leaking a file descriptor each time (the reason Yggdrasil raises `ulimit -n`) and blocking the child forever if it wrote more than the pipe buffer. * Populate `BinaryBuilderBase.all_compiler_shards()` before spawning tasks. Its lazy initialisation is not thread-safe; with a cold cache, 13 of 16 audit tasks failed with "Impossible compiler constraints ... upon GCCBuild[]" when `check_isa` constructed runners concurrently. Assisted-by: Claude Code (Fable 5.1)
giordano
reviewed
Sep 9, 2026
| all_ok = Threads.Atomic{Bool}(true) | ||
|
|
||
| # Populate the compiler shard cache before we start spawning tasks: its lazy | ||
| # initialisation in BinaryBuilderBase is not thread-safe, and every sandboxed |
Member
There was a problem hiding this comment.
That contradicts JuliaPackaging/BinaryBuilderBase.jl#495
giordano
reviewed
Sep 9, 2026
| p = open(`$(Base.julia_cmd()) -e $dlopen_cmd`) | ||
| wait(p) | ||
| if p.exitcode != 0 | ||
| # Discard stdout: `open(cmd)` would leave an unread pipe open per library, |
Member
There was a problem hiding this comment.
So we'll never know why a library couldn't be dlopened? That's very useful information
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Yggdrasil runs the auditor with
--threads 16on Julia 1.12 and has seen audits stall indefinitely (e.g. during the Clang build, last output being theSkipping binary analysis of tools/...lines), something never observed on Julia 1.7. I could not reproduce a deadlock locally (seven runs of the realauditon 118 binaries with 16 threads on 1.12.7 all completed), but reviewing the threaded paths turned up three concrete problems, fixed here.Serial per-library loop in
check_dynamic_linkageEvery iteration of that loop modifies the same binary and serialises on its patchelf lock, so the nested
Threads.@threadsgained nothing. On Julia 1.7 nested@threadsloops ran serially anyway; since Julia 1.9 they spawnnthreads()tasks per outer task, so on CI up to 256 tasks contended on locks and subprocess pipes, each pinned to its thread once it spawned a process with anIOBufferstdout (that goes through@async). This is the only behavioural difference between 1.7 and 1.12 on this code path, so this change restores the 1.7 schedule.dlopenprobe no longer leaks a pipeopen(cmd)followed bywait(p)left an unread stdout pipe open per shared library. That leaks a file descriptor each time (the reason Yggdrasil'sbuild.shraisesulimit -n), and a child writing more than the pipe buffer to stdout would block forever. The probe now runs withstdin/stdoutset todevnulland checkssuccess; stderr is still inherited for verbose error display.Warm the compiler shard cache before spawning tasks
BinaryBuilderBase.all_compiler_shards()lazily assigns an empty vector and then pushes into it, which is not thread-safe. With a cold cache, 13 of 16 audit tasks failed withImpossible compiler constraints ... upon GCCBuild[]whencheck_isaconstructed runners concurrently. Inautobuildthe build warms the cache first, so this only bites standaloneauditcalls, but it is cheap to guard. The proper fix belongs in BinaryBuilderBase.Testing
Verified on Julia 1.12.7: package loads;
check_dynamic_linkagewithautofix=trueruns on real libraries with 8 threads; the new probe succeeds for a child writing 100 kB to stdout (the old code would block there) and reports a child exiting with status 1.🤖 Generated with Claude Code