Hi. I'm trying to run DLRM with 2 nodes each has one GPU. I use NCCL and a simple test works fine. However with DLRM I get the following issue:
[2026-03-09 23:44:20] node30:11193:11203 [0] transport/net_ib.cc:2453 NCCL WARN NET/IB: Got completion from peer 128.138.232.77<32911> with status=12 opcode=0 len=0 vendor err 129 (Send) localGid ::ffff:192.168.1.30 remoteGids::ffff:192.168.1.32 hca mlx5_0
node30:11193:11203 [0] NCCL INFO transport/net.cc:1227 -> 6
[2026-03-09 23:44:20] node30:11193:11203 [0] transport/net_ib.cc:2453 NCCL WARN NET/IB: Got completion from peer 128.138.232.77<39137> with status=12 opcode=0 len=0 vendor err 129 (Send) localGid ::ffff:192.168.1.30 remoteGids::ffff:192.168.1.32 hca mlx5_0
node30:11193:11203 [0] NCCL INFO transport/net.cc:1227 -> 6
I use A2 GPUs over ROCe. My command:
torchrun --nproc_per_node=1 --nnodes=2 --node_rank=0 --master_addr="192.168.1.30" --master_port=1234 dlrm_s_pytorch.py --arch-embedding-size="80000-80000-80000-80000-80000-80000-80000-80000" --arch-sparse-feature-size=128 --arch-mlp-bot="128-128-128-128" --arch-mlp-top="512-512-512-256-1" --max-ind-range=40000000 --data-generation=random --loss-function=bce --round-targets=True --learning-rate=1.0 --mini-batch-size=2048 --print-freq=2 --print-time --test-freq=2 --test-mini-batch-size=2048 --memory-map --use-gpu --num-batches=100 --dist-backend=nccl
and node_rank=1 for the worker.
Hi. I'm trying to run DLRM with 2 nodes each has one GPU. I use NCCL and a simple test works fine. However with DLRM I get the following issue:
I use A2 GPUs over ROCe. My command:
and
node_rank=1for the worker.