From the beta.41 pre-deploy deep audit. Live cluster GPU/VRAM accounting has several correctness gaps (not deploy-blocking for a single-node Pi with no active GPU workers, but should be fixed before the cluster GPU path is leaned on).
H1 - claim_lease over-commits worker VRAM. cluster/manager.py:428-437 checks required_vram_mb > worker.free_vram_mb using stale heartbeat data, never decremented on grant, and does not subtract other active leases on the same worker. Two concurrent 6GB claims against an 8GB-free worker both pass -> OOM/Xid-62 (the class #1859 targets). Fix: subtract the worker's active leases inside the _lease_lock check.
H2 - worker offline/unregister releases leases but never cancels running arbiter tasks. manager.py:309-338 (unregister) + :727-735 (heartbeat-timeout offline) pop leases without cancel_running_for_leases(), unlike the stale-drain (:777-786) and force-drain (:540-547) paths. Orphan task keeps the VRAM reservation + _running slot; a returning worker lets a new claimant run concurrently with the orphan. Fix: mirror the stale-drain wiring in both paths.
M1 - renew() resurrects an expired lease. scheduling/leases.py:159-173 SELECT has no expires_at > now filter and no cleanup inside BEGIN IMMEDIATE, so a lapsed holder that renews before any purge gets a fresh TTL, beating a competing acquire. (Rest of #1900 is correct.) Fix: add the expiry predicate inside the txn.
M2 - second VRAM source of truth, fail-open ~1TB. scheduler/discovery.py:216-218 _gpu_vram_probe returns 999_999 on probe failure and ignores VramReservationManager's in-flight reservations; scheduler/resource.py:154-156 admits against it. Contradicts #1859's single VRAM authority. Fix: route through the shared ledger, fail closed.
M3 - thread-unsafe reservation sweep. gpu_arbiter.py:265 runs available_vram via asyncio.to_thread, but _sweep_stale_unlocked (vram_reservation.py:251-260) mutates _pending/_reserved_vram_mb with no thread lock -> dict-changed-during-iteration / lost update -> phantom or negative reserved VRAM (permanent false load denials). available_vram() is also called synchronously on the loop from routes/models.py:413,674. Fix: probe in thread, account on loop (or add a threading.Lock).
From the beta.41 pre-deploy deep audit. Live cluster GPU/VRAM accounting has several correctness gaps (not deploy-blocking for a single-node Pi with no active GPU workers, but should be fixed before the cluster GPU path is leaned on).
H1 - claim_lease over-commits worker VRAM. cluster/manager.py:428-437 checks required_vram_mb > worker.free_vram_mb using stale heartbeat data, never decremented on grant, and does not subtract other active leases on the same worker. Two concurrent 6GB claims against an 8GB-free worker both pass -> OOM/Xid-62 (the class #1859 targets). Fix: subtract the worker's active leases inside the _lease_lock check.
H2 - worker offline/unregister releases leases but never cancels running arbiter tasks. manager.py:309-338 (unregister) + :727-735 (heartbeat-timeout offline) pop leases without cancel_running_for_leases(), unlike the stale-drain (:777-786) and force-drain (:540-547) paths. Orphan task keeps the VRAM reservation + _running slot; a returning worker lets a new claimant run concurrently with the orphan. Fix: mirror the stale-drain wiring in both paths.
M1 - renew() resurrects an expired lease. scheduling/leases.py:159-173 SELECT has no expires_at > now filter and no cleanup inside BEGIN IMMEDIATE, so a lapsed holder that renews before any purge gets a fresh TTL, beating a competing acquire. (Rest of #1900 is correct.) Fix: add the expiry predicate inside the txn.
M2 - second VRAM source of truth, fail-open ~1TB. scheduler/discovery.py:216-218 _gpu_vram_probe returns 999_999 on probe failure and ignores VramReservationManager's in-flight reservations; scheduler/resource.py:154-156 admits against it. Contradicts #1859's single VRAM authority. Fix: route through the shared ledger, fail closed.
M3 - thread-unsafe reservation sweep. gpu_arbiter.py:265 runs available_vram via asyncio.to_thread, but _sweep_stale_unlocked (vram_reservation.py:251-260) mutates _pending/_reserved_vram_mb with no thread lock -> dict-changed-during-iteration / lost update -> phantom or negative reserved VRAM (permanent false load denials). available_vram() is also called synchronously on the loop from routes/models.py:413,674. Fix: probe in thread, account on loop (or add a threading.Lock).