See included reproducer for the AURORA (PVC) system - Random hangs when using multiple threads with GPU memory and all_to_all collective communication.
Each rank has two std::threads, each thread creates a unique sycl queue (to the same GPU device), allocates unique device memory (malloc_device), uses a unique subcommunicator (with all ranks populated), and then the threads perform MPI all_to_all communication in parallel several times.
Even with my simple test code, two nodes (24 ranks, 12 ppn) randomly hangs - even with the MPI PROGRESS_THROTTLE environment variables set.
To build:
mpicxx -fsycl -lpthread -O2 two_subcomms_alltoall_sycl.cpp
To run (on two nodes, already allocated interactively from PBS):
mpiexec -np 24 -ppn 12 gpu_tile_compact.sh ./a.out
See included reproducer for the AURORA (PVC) system - Random hangs when using multiple threads with GPU memory and all_to_all collective communication.
Each rank has two std::threads, each thread creates a unique sycl queue (to the same GPU device), allocates unique device memory (malloc_device), uses a unique subcommunicator (with all ranks populated), and then the threads perform MPI all_to_all communication in parallel several times.
Even with my simple test code, two nodes (24 ranks, 12 ppn) randomly hangs - even with the MPI PROGRESS_THROTTLE environment variables set.
To build:
mpicxx -fsycl -lpthread -O2 two_subcomms_alltoall_sycl.cpp
To run (on two nodes, already allocated interactively from PBS):
mpiexec -np 24 -ppn 12 gpu_tile_compact.sh ./a.out