python3Packages.torch-bin: fix Python dependencies issues - #123
Merged
Conversation
Python packages in Nix do keep their wheel metadata. This metadata is not used when the packages are used normally in Nix. But the metadata is used by e.g. `pip` in a venv with Nix-managed packages. The torch-bin derivations copy the metadata from the original wheel. The original Torch wheels have dependencies on CUDA/OneAPI packaged as Python packages. This becomes an issue when we want to use Torch inside a venv, such as kernel-builder development shells. When we `pip install` something (like a kernel), pip realizes that the dependencies are missing and attempts to install CUDA/OneAPI. However, this is both unnecessary (Nix already provides these frameworks) and typically fails. This PR addresses this issue by specifying what dependencies should be removed from the metadata. Nix has the `pythonRelaxDepsHook` hook for this. However it fires to early, because it is made for from-source builds. So we vendor this hook as `pythonRelaxWheelDepsHook`. The new hook only addresses part of the problem, since new dependencies might be added in the future, and if we miss them we'll silently have the same issue again. So in addition to removing unneeded dependencies, we also want to make sure that all remaining dependencies are satisfied. To do this, we vendor the `pythonRuntimeDepsCheckHook` as `pythonWheelDepsCheckHook` and modify it to work with binary wheels. With this hook set up, we will be warned of future dependency additions (both needed and unneeded).
danieldk
commented
Oct 27, 2025
| @@ -0,0 +1,93 @@ | |||
| # shellcheck shell=bash | |||
Member
Author
There was a problem hiding this comment.
These scripts come from nixpkgs with some small changes to work with binary packages.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python packages in Nix do keep their wheel metadata. This metadata is not used when the packages are used normally in Nix. But the metadata is used by e.g.
pipin a venv with Nix-managed packages.The torch-bin derivations copy the metadata from the original wheel. The original Torch wheels have dependencies on CUDA/OneAPI packaged as Python packages. This becomes an issue when we want to use Torch inside a venv, such as kernel-builder development shells. When we
pip installsomething (like a kernel), pip realizes that the dependencies are missing and attempts to install CUDA/OneAPI. However, this is both unnecessary (Nix already provides these frameworks) and typically fails.This PR addresses this issue by specifying what dependencies should be removed from the metadata. Nix has the
pythonRelaxDepsHookhook for this. However it fires to early, because it is made for from-source builds. So we vendor this hook aspythonRelaxWheelDepsHook.The new hook only addresses part of the problem, since new dependencies might be added in the future, and if we miss them we'll silently have the same issue again. So in addition to removing unneeded dependencies, we also want to make sure that all remaining dependencies are satisfied. To do this, we vendor the
pythonRuntimeDepsCheckHookaspythonWheelDepsCheckHookand modify it to work with binary wheels. With this hook set up, we will be warned of future dependency additions (both needed and unneeded).