Problem Description
When a GPU request cannot be satisfied against the free capacity being considered, build_node_allocation still inserts a "gpu" key mapped to an empty vector. That leaves a ResourceAllocations in an inconsistent state:
is_empty() inspects the device map and reports "not empty" (a key is present).has_devices() inspects inside the map and reports "no devices" (the list is empty).
The two predicates then disagree about the same allocation. This empty-vector state is reachable in production: a job deferred to a future start whose GPUs are all currently busy resolves its device pick against zero free GPUs and books {"gpu": []}.
Expected: is_empty() and has_devices() always agree on whether an
allocation holds any devices, and a job deferred to a future window is never
booked holding fewer GPUs than it requested.
Environment
- OS: Ubuntu 22.04 LTS
- CPU: AMD EPYC (x86_64 server-class)
- GPU: AMD Instinct MI300X
- ROCm Version: N/A — issue is in the spur scheduler/CLI, independent of the ROCm runtime
- Component: spur
Steps to Reproduce
On a node whose only GPU is already allocated, call build_node_allocation for a GPU request: the returned allocation contains a "gpu" key with an empty list, is_empty() returns false while has_devices() returns false.
Additional Information
Impact: callers that branch on is_empty() vs has_devices() see contradictory answers, and a deferred GPU job can be booked holding none of the GPUs it needs — letting another job take the same window.
Problem Description
When a GPU request cannot be satisfied against the free capacity being considered,
build_node_allocationstill inserts a"gpu"key mapped to an empty vector. That leaves aResourceAllocationsin an inconsistent state:is_empty()inspects the device map and reports "not empty" (a key is present).has_devices()inspects inside the map and reports "no devices" (the list is empty).The two predicates then disagree about the same allocation. This empty-vector state is reachable in production: a job deferred to a future start whose GPUs are all currently busy resolves its device pick against zero free GPUs and books
{"gpu": []}.Expected:
is_empty()andhas_devices()always agree on whether anallocation holds any devices, and a job deferred to a future window is never
booked holding fewer GPUs than it requested.
Environment
Steps to Reproduce
On a node whose only GPU is already allocated, call
build_node_allocationfor a GPU request: the returned allocation contains a"gpu"key with an empty list,is_empty()returns false whilehas_devices()returns false.Additional Information
Impact: callers that branch on
is_empty()vshas_devices()see contradictory answers, and a deferred GPU job can be booked holding none of the GPUs it needs — letting another job take the same window.