installation issue: undefined symbol: iJITNotifyEvent

[cs-mshah]

Thanks for the amazing work. While installing the packages and creating the environment, I encountered an issue when installing gsplat.

[fanegg]

Hi @cs-mshah, thanks for your interest in our work. It looks like the error comes from a mismatch between PyTorch and Python / GLIBC / CUDA driver versions; please install a PyTorch that matches your Python and CUDA setup.

[Sjeary]

torch 2.5.1 and cuda12.4 is avaliavle

[cs-mshah]

I have torch 2.5.1 and CUDA as 12.4 when I do torch.version.cuda. I get the following error now:

File "/data/manan/Human3R/src/dust3r/blocks.py", line 309, in forward
    cross_attn_output, cross_attn = self.cross_attn(self.norm2(x), y_, y_, xpos, ypos, use_ttt3r=use_ttt3r)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/manan/Human3R/src/dust3r/blocks.py", line 235, in forward
    attn = (q @ k.transpose(-2, -1)) * self.scale  # [B, num_heads, Nq, Nk] [1, 16, 768, 1 + 576]
            ~~^~~~~~~~~~~~~~~~~~~~~
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &be
ta, c, ldc, stridec, num_batches)`

[YellowThree-HS]

I have torch 2.5.1 and CUDA as 12.4 when I do torch.version.cuda. I get the following error now:

File "/data/manan/Human3R/src/dust3r/blocks.py", line 309, in forward
cross_attn_output, cross_attn = self.cross_attn(self.norm2(x), y_, y_, xpos, ypos, use_ttt3r=use_ttt3r)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/manan/Human3R/src/dust3r/blocks.py", line 235, in forward
attn = (q @ k.transpose(-2, -1)) * self.scale # [B, num_heads, Nq, Nk] [1, 16, 768, 1 + 576]
~~^~~~~~~~~~~~~~~~~~~~~
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &be ta, c, ldc, stridec, num_batches)

我也遇到这个问题了

[Sjeary]

conda install pytorch torchvision pytorch-cuda=12.4 -c pytorch -c nvidia
replace with
pip install --index-url https://download.pytorch.org/whl/cu124 torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1

把readme中的conda install pytorch torchvision pytorch-cuda=12.4 -c pytorch -c nvidia
替换成为
pip install --index-url https://download.pytorch.org/whl/cu124 torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1
我是这样解决的

[1nnoh]

Way to fix it：

# Downgrade MKL/OpenMP to a known-good set
conda install -y "mkl<2024.1" "intel-openmp<2024.1"

from https://pro2017001.medium.com/installing-pytorch-cuda-11-8-and-pytorch-on-python-3-11-1fe872f29368

[fanegg]

Thanks for the amazing work. While installing the packages and creating the environment, I encountered an issue when installing gsplat.

The undefined symbol iJIT_NotifyEvent is typically caused by MKL version incompatibility: MKL>2024.0 removed this symbol, while PyTorch still depends on it.

Solution:

1. Check your current MKL version:

   conda list mkl
   

2. After installing PyTorch, downgrade MKL to 2024.0.0 or an earlier version:

   conda install mkl=2024.0.0 mkl-service mkl_fft mkl_random -c conda-forge -y
   

3. Verify the fix:

   python -c "import torch; print(torch.__version__)"
   

That worked for me.

[yqytomorrow]

I have torch 2.5.1 and CUDA as 12.4 when I do torch.version.cuda. I get the following error now:
File "/data/manan/Human3R/src/dust3r/blocks.py", line 309, in forward
cross_attn_output, cross_attn = self.cross_attn(self.norm2(x), y_, y_, xpos, ypos, use_ttt3r=use_ttt3r)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/human3r/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/manan/Human3R/src/dust3r/blocks.py", line 235, in forward
attn = (q @ k.transpose(-2, -1)) * self.scale # [B, num_heads, Nq, Nk] [1, 16, 768, 1 + 576]
~~^~~~~~~~~~~~~~~~~~~~~
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &be ta, c, ldc, stridec, num_batches)

我也遇到这个问题了

我也遇到了同样的问题，请问你是如何解决这个问题的？