Skip to content

GPU/cluster VRAM + lease correctness (beta.41 Fable audit: H1/H2/M1/M2/M3) #1992

Description

@jaylfc

From the beta.41 pre-deploy deep audit. Live cluster GPU/VRAM accounting has several correctness gaps (not deploy-blocking for a single-node Pi with no active GPU workers, but should be fixed before the cluster GPU path is leaned on).

H1 - claim_lease over-commits worker VRAM. cluster/manager.py:428-437 checks required_vram_mb > worker.free_vram_mb using stale heartbeat data, never decremented on grant, and does not subtract other active leases on the same worker. Two concurrent 6GB claims against an 8GB-free worker both pass -> OOM/Xid-62 (the class #1859 targets). Fix: subtract the worker's active leases inside the _lease_lock check.

H2 - worker offline/unregister releases leases but never cancels running arbiter tasks. manager.py:309-338 (unregister) + :727-735 (heartbeat-timeout offline) pop leases without cancel_running_for_leases(), unlike the stale-drain (:777-786) and force-drain (:540-547) paths. Orphan task keeps the VRAM reservation + _running slot; a returning worker lets a new claimant run concurrently with the orphan. Fix: mirror the stale-drain wiring in both paths.

M1 - renew() resurrects an expired lease. scheduling/leases.py:159-173 SELECT has no expires_at > now filter and no cleanup inside BEGIN IMMEDIATE, so a lapsed holder that renews before any purge gets a fresh TTL, beating a competing acquire. (Rest of #1900 is correct.) Fix: add the expiry predicate inside the txn.

M2 - second VRAM source of truth, fail-open ~1TB. scheduler/discovery.py:216-218 _gpu_vram_probe returns 999_999 on probe failure and ignores VramReservationManager's in-flight reservations; scheduler/resource.py:154-156 admits against it. Contradicts #1859's single VRAM authority. Fix: route through the shared ledger, fail closed.

M3 - thread-unsafe reservation sweep. gpu_arbiter.py:265 runs available_vram via asyncio.to_thread, but _sweep_stale_unlocked (vram_reservation.py:251-260) mutates _pending/_reserved_vram_mb with no thread lock -> dict-changed-during-iteration / lost update -> phantom or negative reserved VRAM (permanent false load denials). available_vram() is also called synchronously on the loop from routes/models.py:413,674. Fix: probe in thread, account on loop (or add a threading.Lock).

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingclusterDistributed cluster and workersfork-auditTracking issue for fork-vs-upstream drift auditskilo-auto-fixAuto-generated label by Kilokilo-triagedAuto-generated label by KiloschedulerResource scheduler

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions