The shared-memory hash groupby can hang forever when the host-side compatibility check passes but the device-side packing cannot fit all aggregation columns in one pass. This is a follow-on to #17853: the guard added by PR #17851 checks only the single-column data footprint, while the kernel needs room for every column's data plus its validity mask plus 16-byte alignment padding.
Mechanism
is_shared_memory_compatible (cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36) accepts the shared-memory path when data_buffer_size >= size * GROUPBY_CARDINALITY_THRESHOLD for each column individually. The kernel's packing step calculate_columns_to_aggregate (cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153) instead needs, for all columns together, sum_i(round_up16(width_i*C) + round_up16(C)) bytes where C is the block cardinality (up to 128). When that sum does not fit, the loop breaks without advancing and the driver while (col_end < num_cols) at line 302 has no progress condition, so it re-runs the identical break point forever; any single spinning block hangs the whole kernel.
For int64 aggregations at cardinality 128 one column needs round16(8*128) + round16(128) = 1152 bytes, but the guard only requires avail - offsets >= 1024. Any configuration with per-column slack in [1024, 1152) passes the host check and then spins.
Reproduced on RTX 5060 Ti (sm_120), libcudf 26.10 nightly
One int64 key column with all-distinct keys plus k int64 value columns aggregated with SUM:
| rows |
k (number of value columns) |
| 27648 |
868-870 hang, neighbors ok |
| 23040 |
1082-1086 hang, neighbors ok |
| 18432 |
1402-1406 hang, neighbors ok |
| 13824 |
1934 hangs |
Each hanging config deterministically times out (60 s); configs just outside the narrow windows return normally, matching the computed window boundary exactly. A cuda-gdb backtrace of a live hang shows the main thread blocked in cudaFree inside compute_single_pass_aggs, i.e. the launched single_pass_shmem_aggs_kernel never retires.
The same window arithmetic predicts hangs on H100/B200 at different occupancy points (for example 6 int64 aggregations at blocksPerSM=16 on 228KB parts).
Suggested fix
Tighten the host check to the true worst-case total the packer needs at cardinality 128: require sum_i(round_up16(width_i*128) + round_up16(128)) <= data_buffer_size (falling back to the existing global-memory path otherwise). That is conservative for blocks whose cardinality is below 128 but eliminates the spin.
The shared-memory hash groupby can hang forever when the host-side compatibility check passes but the device-side packing cannot fit all aggregation columns in one pass. This is a follow-on to #17853: the guard added by PR #17851 checks only the single-column data footprint, while the kernel needs room for every column's data plus its validity mask plus 16-byte alignment padding.
Mechanism
is_shared_memory_compatible(cpp/src/groupby/hash/compute_single_pass_aggs.cu:26-36) accepts the shared-memory path whendata_buffer_size >= size * GROUPBY_CARDINALITY_THRESHOLDfor each column individually. The kernel's packing stepcalculate_columns_to_aggregate(cpp/src/groupby/hash/compute_shared_memory_aggs.cu:121-153) instead needs, for all columns together,sum_i(round_up16(width_i*C) + round_up16(C))bytes where C is the block cardinality (up to 128). When that sum does not fit, the loop breaks without advancing and the driverwhile (col_end < num_cols)at line 302 has no progress condition, so it re-runs the identical break point forever; any single spinning block hangs the whole kernel.For int64 aggregations at cardinality 128 one column needs
round16(8*128) + round16(128) = 1152bytes, but the guard only requiresavail - offsets >= 1024. Any configuration with per-column slack in[1024, 1152)passes the host check and then spins.Reproduced on RTX 5060 Ti (sm_120), libcudf 26.10 nightly
One int64 key column with all-distinct keys plus k int64 value columns aggregated with SUM:
Each hanging config deterministically times out (60 s); configs just outside the narrow windows return normally, matching the computed window boundary exactly. A cuda-gdb backtrace of a live hang shows the main thread blocked in
cudaFreeinsidecompute_single_pass_aggs, i.e. the launchedsingle_pass_shmem_aggs_kernelnever retires.The same window arithmetic predicts hangs on H100/B200 at different occupancy points (for example 6 int64 aggregations at blocksPerSM=16 on 228KB parts).
Suggested fix
Tighten the host check to the true worst-case total the packer needs at cardinality 128: require
sum_i(round_up16(width_i*128) + round_up16(128)) <= data_buffer_size(falling back to the existing global-memory path otherwise). That is conservative for blocks whose cardinality is below 128 but eliminates the spin.