[R9700][gfx1201] System-wide soft lockup in amdttm_resource_manager_usage triggered by concurrent nvtop and rocm-smi telemetry polling

[tamascode]

[R9700][gfx1201] System-wide soft lockup in amdttm_resource_manager_usage triggered by concurrent nvtop and rocm-smi telemetry polling

Description

My Ubuntu server hard-locks while monitoring four AMD Radeon AI PRO R9700 GPUs.

The issue occurred while nvtop was running and a Python health-check script executed:

rocm-smi --showuse --showmemuse --showtemp --json

The server became completely unresponsive and required a reboot.

The kernel watchdog reported multiple CPU soft lockups. Every affected process was blocked in the AMD TTM VRAM accounting path:

native_queued_spin_lock_slowpath
_raw_spin_lock
amdttm_resource_manager_usage
amdgpu_mem_info_vram_used_show
dev_attr_show
sysfs_kf_seq_show
kernfs_seq_show
seq_read_iter
kernfs_fop_read_iter
vfs_read
ksys_read

This appears to be a deadlock in the AMDGPU/TTM driver while GPU telemetry tools concurrently read VRAM usage.

A userspace telemetry query should not be able to deadlock the kernel or freeze the entire system.

Hardware

CPU: AMD Threadripper platform
Motherboard: ASRock X399 Professional Gaming
BIOS: P3.30, 2018-08-14
GPUs: 4 x AMD Radeon AI PRO R9700
GPU architecture: gfx1201

Software

Operating system: Ubuntu 24.04
Kernel at time of failure: 6.17.0-35-generic
AMDGPU module version reported by kernel: 6.19.0
Driver type: externally installed DKMS/out-of-tree AMDGPU module

Monitoring applications active at the time:

nvtop
rocm-smi

The health-check script called:

rocm-smi --showuse --showmemuse --showtemp --json

Steps to reproduce

1. Use a system with four Radeon AI PRO R9700 GPUs.
2. Start nvtop.
3. While nvtop is running, execute:

rocm-smi --showuse --showmemuse --showtemp --json

4. The system may become unresponsive.
5. Kernel watchdog messages show multiple CPUs blocked in:

amdttm_resource_manager_usage
amdgpu_mem_info_vram_used_show

The issue happened more than once. Running GPU telemetry commands appears sufficient to trigger the failing path.

Expected behavior

nvtop and rocm-smi should be able to query utilization, temperature, and VRAM information concurrently without affecting system stability.

Actual behavior

The AMDGPU kernel driver appears to deadlock while reading VRAM usage. Multiple userspace telemetry processes block on the same kernel spinlock, several CPUs report soft lockups, and the entire server becomes unusable.

Representative kernel output

watchdog: BUG: soft lockup - CPU#6 stuck for 336s! [python3:127428]

CPU: 6 PID: 127428 Comm: python3
Tainted: G D O E L
6.17.0-35-generic

RIP: native_queued_spin_lock_slowpath

Call Trace:
 _raw_spin_lock
 amdttm_resource_manager_usage [amdttm]
 amdgpu_mem_info_vram_used_show [amdgpu]
 dev_attr_show
 sysfs_kf_seq_show
 kernfs_seq_show
 seq_read_iter
 kernfs_fop_read_iter
 vfs_read
 ksys_read
 __x64_sys_read

Additional CPUs were reported stuck for long periods:

CPU#12 stuck for 727 seconds
CPU#21 stuck for 1065 seconds
CPU#22 stuck for 674 seconds
CPU#28 stuck for 283 seconds

Additional observations

• No OOM event was identified.
• No RAID or disk fault appears to be the cause.
• The repeated common failure point is AMDGPU VRAM accounting.
• nvtop was continuously displaying VRAM usage.
• rocm-smi --showmemuse queried VRAM usage at the same time.
• The system required a forced reboot.
• I have disabled GPU telemetry collection to avoid further lockups.
• The issue reproduced again when running an AMD GPU information query, confirming that telemetry access can trigger the deadlock.

Requested investigation

Please investigate whether the TTM resource-manager lock can remain held during concurrent reads of:

/sys/class/drm/card*/device/mem_info_vram_used

Please also verify whether concurrent VRAM telemetry polling from nvtop, rocm-smi, or amd-smi can deadlock against another AMDGPU memory-management operation on gfx1201/R9700 systems.

Attachments

Attach the complete kernel crash log:

crashfile.txt

Before uploading, remove any internal IP addresses, usernames, Slack webhook information, or unrelated journal entries.

[darren-amd]

Hi @tamascode,

I gave this a try locally and was unable to reproduce the issue. Could you please try upgrading amdgpu, and if the issue persists could you attach the kernel log that you mentioned?

[tamascode]

@darren-amd, thanks for taking a look. I found that this happens when the GPUs are close to using all available VRAM. Once memory usage approaches the limit, the issue appears.

As a temporary workaround, I need to keep VRAM usage below 31 GB per GPU.