feat(cluster): expose real-time free/used VRAM in /api/cluster/workers#1673
feat(cluster): expose real-time free/used VRAM in /api/cluster/workers#1673hognek wants to merge 2 commits into
Conversation
Add model archive promotion engine: when a new worker registers with hardware that can run previously-archived models, automatically promote them from ~/taos/archive/models/ to the active models tree. - model_archive.py: archive scan, hardware compatibility check, model promotion (move + manifest cleanup), notifications - ClusterManager.register_worker: auto-trigger promotion on worker join - GET /api/cluster/promote-archived: manual promotion scan endpoint - 29 unit tests covering compatibility checks, archive I/O, promotable filtering, and file movement
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
|
👋 Thanks for the PR! This one targets See CONTRIBUTING.md for the branch model. |
📝 WalkthroughWalkthroughThis PR adds a model archive promotion system that scans a manifest-based archive, checks worker hardware compatibility, and moves promotable model files into the active tree during worker registration or via a new admin endpoint. It also adds GPU VRAM tracking fields propagated through worker heartbeats. ChangesModel archive promotion and VRAM reporting
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant WorkerAgent
participant ClusterRoute
participant ClusterManager
participant ModelArchive
participant FileSystem
participant Notifications
WorkerAgent->>ClusterRoute: POST heartbeat(free_vram_mb, used_vram_mb)
ClusterRoute->>ClusterManager: heartbeat(..., free_vram_mb, used_vram_mb)
ClusterManager->>ClusterManager: store VRAM values on worker
WorkerAgent->>ClusterManager: register_worker(info)
ClusterManager->>ModelArchive: promote_compatible_models(hardware, name)
ModelArchive->>FileSystem: list_archived_models()
ModelArchive->>ModelArchive: _worker_can_run() check per manifest
ModelArchive->>FileSystem: promote_model() moves files, removes manifest
ModelArchive->>Notifications: emit_event("model.promoted")
ModelArchive-->>ClusterManager: promoted model ids
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![kilo-code-bot[bot]](https://avatars.githubusercontent.com/in/2193792?s=60&v=4){kind=small}
| req_npu = requirements.get("npu_type") | ||
| if req_npu: | ||
| worker_npu_type = hw_npu.get("type", "none") or "none" | ||
| if worker_npu_type != req_npu and worker_npu_type not in (req_npu,): |
There was a problem hiding this comment.
WARNING: Dead/redundant clause in NPU check.
if worker_npu_type != req_npu and worker_npu_type not in (req_npu,):worker_npu_type not in (req_npu,) is logically identical to worker_npu_type != req_npu (a 1-tuple membership test equals equality on its single element). It is dead code that obscures intent and should be removed.
| if worker_npu_type != req_npu and worker_npu_type not in (req_npu,): | |
| if worker_npu_type != req_npu: | |
| return False |
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| level="info", | ||
| ) | ||
|
|
||
| # Promote any archived models this worker can now run |
There was a problem hiding this comment.
WARNING: promote_compatible_models is invoked from an async def but runs blocking filesystem I/O (Path.is_dir, Path.glob, json.loads, shutil.move) on the event loop.
register_worker is async def and promote_compatible_models is async, but its body delegates to find_promotable -> list_archived_models (glob + read every manifest) and promote_model (shutil.move + unlink) — all synchronous, all blocking. With a non-trivial archive and large model files this will stall the controller's event loop and delay heartbeats, other registrations, and request handling.
Recommend wrapping the scan and move in asyncio.to_thread(...) (or loop.run_in_executor) and keeping only the notifications.emit_event awaited.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| promoted_by_worker: dict[str, list[str]] = {} | ||
| total = 0 | ||
|
|
||
| for w in online: |
There was a problem hiding this comment.
WARNING: find_promotable / promote_model run synchronously inside an async def route, performing O(online_workers * archived_models) filesystem reads plus shutil.move of model files.
promote_archived_models loops over every online worker, and each iteration calls find_promotable (which re-reads every manifest) followed by promote_model (shutil.move of model files) inline in the async handler. A single request can therefore block the event loop for the duration of multi-GB copies and time out under realistic loads.
Recommend offloading archive scan + movement to await asyncio.to_thread(...) and deduping list_archived_models across workers (e.g. read the archive once, then per-worker filter).
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| return JSONResponse({"error": str(exc)}, status_code=502) | ||
|
|
||
|
|
||
| @router.get("/api/cluster/promote-archived") |
There was a problem hiding this comment.
SUGGESTION: The new /api/cluster/promote-archived endpoint appears to be unauthenticated, while it performs filesystem-level shutil.move operations and emits notifications.
Other endpoints in this file are gated by a shared/worker token; adding an un-authenticated route that can move files into the active models tree and emit admin-visible notifications expands the cluster's external attack surface. Consider gating this route behind the same auth scheme used by sibling endpoints (or at minimum an admin token), even though it's "intended" for the admin UI.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| """ | ||
| promotable: list[dict] = [] | ||
| for model in list_archived_models(archive_dir): | ||
| reqs = model.get("requirements") or {} |
There was a problem hiding this comment.
WARNING: find_promotable mutates the manifest dict in place by assigning model["worker_name"] = worker_name.
Today this is "safe" only because list_archived_models rebuilds each dict from JSON every call. The instant anyone caches list_archived_models results (an obvious perf win given the per-call I/O cost), the same dict reference gets worker_name re-stamped across calls and promote_model will silently mis-attribute the promotion. Make ownership unambiguous with a copy:
| reqs = model.get("requirements") or {} | |
| promotable.append({**model, "worker_name": worker_name}) |
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
|
|
||
| if not model_files_dir.is_dir(): | ||
| logger.warning( | ||
| "model_archive: model files directory %s not found — " |
There was a problem hiding this comment.
WARNING: promote_model deletes the archive manifest before confirming the archive files are moved, and is not crash-safe.
When target_dir.exists(), the code unlink()s the manifest while leaving the archive files directory completely untouched. If target_dir is a stale empty stub from a previous failure, the model files are silently orphaned in the archive forever (no manifest, no move). More generally, shutil.move then manifest_path.unlink() is not atomic across crashes — a power loss between the two leaves files in archive with no manifest.
Recommend: (1) log a warning when the target pre-existed so the operator knows files were left behind, (2) move manifest deletion into the same try block that already wraps shutil.move, and re-create the manifest on failure, or (3) move into a temp dir, fsync, then atomically rename into place + delete the manifest.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| caps = self.detect_capabilities(backends) | ||
| kv_quant = self.detect_kv_quant_support(backends) | ||
| snap = capacity_snapshot() | ||
| vram = gpu_vram_snapshot() |
There was a problem hiding this comment.
WARNING: Heartbeat now blocks on read_nvidia_vram() (typically a shell-out to nvidia-smi) with no timeout.
gpu_vram_snapshot() is invoked inline in heartbeat(). If the NVIDIA driver is faulted or NVML hangs, the probe stalls until the OS-level subprocess times out (or never). Meanwhile the controller will mark this worker offline and trigger a re-register storm.
The docstring promises "a missing nvidia-smi or a probe timeout is not an error" but no timeout is actually enforced in this PR. Wrap the probe with asyncio.wait_for / subprocess timeout, or run it in asyncio.to_thread with a watchdog.
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
| available or the probe fails. | ||
|
|
||
| Called from the worker heartbeat loop alongside capacity_snapshot(). | ||
| Best-effort: a missing nvidia-smi or a probe timeout is not an error. |
There was a problem hiding this comment.
WARNING: gpu_vram_snapshot exposes VRAM for "the first GPU" only — multi-GPU hosts are silently mis-reported.
read_nvidia_vram() returns (used_mb, total_mb) for GPU 0. The agent heartbeat then publishes this as the worker's total free/used VRAM. On a host where GPU 0 is saturated but GPU 1 is idle, Skald's TaosDispatcher will route to this worker under the false impression it has headroom, and ignore workers that actually do.
Either aggregate across all GPUs (sum free, sum used) and expose the totals, or document the limit and surface free_vram_mb as max across GPUs so the router picks "does this worker have at least one usable card?"
Reply with @kilocode-bot fix it to have Kilo Code address this issue.
hognek
force-pushed
the
feat/vram-realtime-worker-fields
branch
from
64d530b to
893ed7b
Compare
taOS jaylfc#894 slice — additive data, no scheduler behavior change. Worker side (worker/agent.py, worker_capacity.py): - Add gpu_vram_snapshot() to worker_capacity that reads real-time free/used VRAM via system_stats.read_nvidia_vram() - Worker heartbeat now samples VRAM and includes free_vram_mb + used_vram_mb in the heartbeat JSON payload Controller side (routes/cluster.py, manager.py, worker_protocol.py): - WorkerInfo gains free_vram_mb and used_vram_mb fields (default 0) - HeartbeatBody accepts optional free_vram_mb / used_vram_mb - ClusterManager.heartbeat() stores VRAM when provided; absent values leave existing stored values unchanged (legacy compat) - GET /api/cluster/workers surfaces both fields via asdict() Tests (test_routes_cluster.py): - 3 new tests: default-zero, heartbeat updates, legacy-survive Skald's TaosDispatcher._route_taos can now sort candidates by actual free VRAM instead of hardware.gpu.vram_mb (total capacity).
hognek
force-pushed
the
feat/vram-realtime-worker-fields
branch
from
893ed7b to
86d0012
Compare
|
|
| return JSONResponse({"error": str(exc)}, status_code=502) | ||
|
|
||
|
|
||
| @router.get("/api/cluster/promote-archived") |
There was a problem hiding this comment.
WARNING: GET with mutating side effects — violates HTTP semantics and risks duplicate execution.
@router.get("/api/cluster/promote-archived") triggers shutil.move of multi-GB model files and emits notifications. GET requests are expected to be safe and idempotent; intermediaries (CDNs, browsers, proxies, link previews) can cache or prefetch GET responses, potentially re-triggering the promotion and double-emitting model.promoted notifications. Should be @router.post(...) (or PATCH) so the verb signals mutating intent and is not cached/refetched.
| model_id, | ||
| ) | ||
|
|
||
| return { |
There was a problem hiding this comment.
WARNING: Endpoint silently reports success on partial failure — no error visibility for callers.
promote_model() swallows OSError/shutil.Error and returns False, but the route only counts successful promotions and discards failures. A caller cannot tell whether promoted: 0 means "nothing was promotable" vs "everything failed". Consider returning per-worker failure counts (or a failures array with model_id + reason) so the admin UI / CLI can surface actionable errors when a worker should have been able to run a model but the move failed (permissions, disk full, cross-device, etc).
Code Review SummaryStatus: 2 New Issues Found | Recommendation: Address before merge Overview
The 8 issues from the previous review pass (NPU dead clause, blocking I/O on the event loop in Issue Details (click to expand)WARNING
Files Reviewed (8 files)
Fix these issues in Kilo Cloud Reviewed by minimax-m3 · Input: 40.4K · Output: 5.5K · Cached: 245.2K |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (4)
tinyagentos/worker/agent.py (1)
396-404: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBlocking subprocess call on the event loop.
gpu_vram_snapshot()(viaread_nvidia_vram()) usessubprocess.run(..., timeout=3), a blocking call invoked directly in the asyncheartbeat()coroutine. This can stall the event loop for up to 3s per heartbeat cycle (every 5s), delaying any other coroutines sharing this loop (e.g. a local worker HTTP server onworker_port).Offload it to a thread to keep the loop responsive:
♻️ Proposed fix
- snap = capacity_snapshot() - vram = gpu_vram_snapshot() + snap = capacity_snapshot() + vram = await asyncio.to_thread(gpu_vram_snapshot)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tinyagentos/worker/agent.py` around lines 396 - 404, The heartbeat coroutine is calling gpu_vram_snapshot() synchronously, which can block the event loop via read_nvidia_vram() and subprocess.run. Update tinyagentos.worker.agent.Agent.heartbeat to offload the gpu_vram_snapshot() work to a thread (for example alongside the existing capacity_snapshot flow) so the async loop stays responsive; keep the change localized to the load/capacity collection path and preserve the existing backend/capability detection behavior.tinyagentos/cluster/model_archive.py (2)
199-213: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueNon-empty archive files dir is left behind when target already exists.
When
target_direxists you unlink the manifest and onlyrmdirthe files dir if it's empty; a populatedmodel_files_diris silently orphaned in the archive with no manifest referencing it. Considershutil.rmtree(or logging) so stale archive files don't accumulate.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tinyagentos/cluster/model_archive.py` around lines 199 - 213, The target-exists branch in model_archive handling leaves populated archive files behind because it only calls model_files_dir.rmdir() when the directory is empty. Update the existing target_dir.exists() path in model_archive to remove the archive contents more robustly, using shutil.rmtree on model_files_dir or equivalent cleanup, and keep the manifest unlink behavior so stale files do not accumulate.
128-132: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRedundant NPU condition.
worker_npu_type not in (req_npu,)is equivalent toworker_npu_type != req_npu, so theandclause is a tautology.♻️ Simplify
if req_npu: worker_npu_type = hw_npu.get("type", "none") or "none" - if worker_npu_type != req_npu and worker_npu_type not in (req_npu,): + if worker_npu_type != req_npu: return False🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tinyagentos/cluster/model_archive.py` around lines 128 - 132, The NPU compatibility check in model_archive.py is redundant because `worker_npu_type not in (req_npu,)` duplicates the `worker_npu_type != req_npu` comparison. Simplify the `req_npu` branch in `model_archive.py` by keeping a single inequality check in the `worker_npu_type` comparison, preserving the same behavior while removing the tautological `and` condition.tests/test_model_archive.py (1)
140-147: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winApple-silicon test doesn't exercise the unified-memory branch.
With
vram_mb=16384already ≥min_vram_mb=8192, the check passes regardless of themax(worker_vram, hw_ram)logic. To actually cover the unified-memory path, setvram_mbbelow the requirement and rely onram_mbto satisfy it (e.g.vram_mb=0,ram_mb=16384).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_model_archive.py` around lines 140 - 147, The Apple-silicon test currently passes without exercising the unified-memory logic in _worker_can_run. Update test_apple_silicon_unified_memory so the worker GPU vram_mb is below the min_vram_mb requirement and the assertion depends on ram_mb satisfying it via the unified-memory path. Keep the test focused on the aarch64 Apple hardware case and verify that the max(worker_vram, hw_ram) behavior is what makes the result True.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tinyagentos/cluster/model_archive.py`:
- Line 215: Offload the synchronous model promotion work in promote_model() so
it does not block the event loop when called from ClusterManager.register_worker
or GET /api/cluster/promote-archived. Move the filesystem-heavy logic around
shutil.move in model_archive.py into a thread using asyncio.to_thread() or
run_in_executor(), keeping the async callers responsive during large archive
promotions.
In `@tinyagentos/routes/cluster.py`:
- Around line 669-670: The promote_archived_models endpoint is mutating state
but is currently exposed as a GET route, which should be safe and non-mutating.
Update the route decorator on promote_archived_models to use POST instead of
GET, and keep the existing file-moving/manifests-deleting behavior behind that
method so it is not triggerable via browser prefetch or CSRF-prone GET requests.
---
Nitpick comments:
In `@tests/test_model_archive.py`:
- Around line 140-147: The Apple-silicon test currently passes without
exercising the unified-memory logic in _worker_can_run. Update
test_apple_silicon_unified_memory so the worker GPU vram_mb is below the
min_vram_mb requirement and the assertion depends on ram_mb satisfying it via
the unified-memory path. Keep the test focused on the aarch64 Apple hardware
case and verify that the max(worker_vram, hw_ram) behavior is what makes the
result True.
In `@tinyagentos/cluster/model_archive.py`:
- Around line 199-213: The target-exists branch in model_archive handling leaves
populated archive files behind because it only calls model_files_dir.rmdir()
when the directory is empty. Update the existing target_dir.exists() path in
model_archive to remove the archive contents more robustly, using shutil.rmtree
on model_files_dir or equivalent cleanup, and keep the manifest unlink behavior
so stale files do not accumulate.
- Around line 128-132: The NPU compatibility check in model_archive.py is
redundant because `worker_npu_type not in (req_npu,)` duplicates the
`worker_npu_type != req_npu` comparison. Simplify the `req_npu` branch in
`model_archive.py` by keeping a single inequality check in the `worker_npu_type`
comparison, preserving the same behavior while removing the tautological `and`
condition.
In `@tinyagentos/worker/agent.py`:
- Around line 396-404: The heartbeat coroutine is calling gpu_vram_snapshot()
synchronously, which can block the event loop via read_nvidia_vram() and
subprocess.run. Update tinyagentos.worker.agent.Agent.heartbeat to offload the
gpu_vram_snapshot() work to a thread (for example alongside the existing
capacity_snapshot flow) so the async loop stays responsive; keep the change
localized to the load/capacity collection path and preserve the existing
backend/capability detection behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 0c40014f-3a7b-45ea-a11d-dee2dcd23696
📒 Files selected for processing (8)
tests/test_model_archive.pytests/test_routes_cluster.pytinyagentos/cluster/manager.pytinyagentos/cluster/model_archive.pytinyagentos/cluster/worker_capacity.pytinyagentos/cluster/worker_protocol.pytinyagentos/routes/cluster.pytinyagentos/worker/agent.py
| pass | ||
| return True | ||
|
|
||
| shutil.move(str(model_files_dir), str(target_dir)) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== model_archive.py outline ==\n'
ast-grep outline tinyagentos/cluster/model_archive.py --view expanded || true
printf '\n== register_worker / promote-archived references ==\n'
rg -n "promote_model|register_worker|promote-archived|shutil\.move|to_thread|run_in_executor" tinyagentos -S
printf '\n== model_archive.py relevant region ==\n'
sed -n '150,260p' tinyagentos/cluster/model_archive.py | cat -n
printf '\n== cluster manager relevant region ==\n'
sed -n '1,260p' tinyagentos/cluster/cluster_manager.py | cat -n
printf '\n== routes relevant region ==\n'
sed -n '1,260p' tinyagentos/cluster/routes.py | cat -nRepository: jaylfc/taOS
Length of output: 10161
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== cluster-related files ==\n'
git ls-files 'tinyagentos/cluster/*' 'tinyagentos/routes/cluster.py' | sort
printf '\n== manager.py outline ==\n'
ast-grep outline tinyagentos/cluster/manager.py --view expanded || true
printf '\n== cluster.py outline ==\n'
ast-grep outline tinyagentos/routes/cluster.py --view expanded || true
printf '\n== manager.py register_worker region ==\n'
sed -n '1,180p' tinyagentos/cluster/manager.py | cat -n
printf '\n== cluster route promote/register regions ==\n'
sed -n '100,220p' tinyagentos/routes/cluster.py | cat -n
sed -n '660,725p' tinyagentos/routes/cluster.py | cat -nRepository: jaylfc/taOS
Length of output: 22709
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== model_archive.py top ==\n'
sed -n '1,120p' tinyagentos/cluster/model_archive.py | cat -n
printf '\n== root helpers references ==\n'
rg -n "_archive_root|_active_models_root|TAOS|models|archive" tinyagentos/cluster/model_archive.py tinyagentos -S
printf '\n== any custom model root config ==\n'
rg -n "active models|archive root|models root|archive_dir|TAOS_.*MODEL|MODEL.*ROOT|ARCHIVE.*ROOT" tinyagentos -SRepository: jaylfc/taOS
Length of output: 50368
Offload model promotion off the event loop. promote_model() does synchronous filesystem work and is awaited from ClusterManager.register_worker and GET /api/cluster/promote-archived; large archive moves can block worker registration and heartbeats, especially when shutil.move() falls back to copying across filesystems. Wrap the promotion work in asyncio.to_thread()/run_in_executor() from the async path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tinyagentos/cluster/model_archive.py` at line 215, Offload the synchronous
model promotion work in promote_model() so it does not block the event loop when
called from ClusterManager.register_worker or GET /api/cluster/promote-archived.
Move the filesystem-heavy logic around shutil.move in model_archive.py into a
thread using asyncio.to_thread() or run_in_executor(), keeping the async callers
responsive during large archive promotions.
| @router.get("/api/cluster/promote-archived") | ||
| async def promote_archived_models(request: Request): |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
State-mutating operation exposed as GET.
promote-archived moves files and deletes manifests, but GET is expected to be safe/idempotent. Exposing it as GET makes it triggerable by browser prefetch, crawlers, or cross-site requests (CSRF) without a request body. Use @router.post instead.
🔒 Suggested change
-@router.get("/api/cluster/promote-archived")
+@router.post("/api/cluster/promote-archived")
async def promote_archived_models(request: Request):📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| @router.get("/api/cluster/promote-archived") | |
| async def promote_archived_models(request: Request): | |
| `@router.post`("/api/cluster/promote-archived") | |
| async def promote_archived_models(request: Request): |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tinyagentos/routes/cluster.py` around lines 669 - 670, The
promote_archived_models endpoint is mutating state but is currently exposed as a
GET route, which should be safe and non-mutating. Update the route decorator on
promote_archived_models to use POST instead of GET, and keep the existing
file-moving/manifests-deleting behavior behind that method so it is not
triggerable via browser prefetch or CSRF-prone GET requests.
2 participants
taOS #894 slice — additive VRAM fields in worker payload
What: Adds real-time
free_vram_mbandused_vram_mbto each worker inGET /api/cluster/workers. Workers report these on every heartbeat; the controller stores and surfaces them.Why: Skald's
TaosDispatcher._route_taoscurrently sorts candidates byhardware.gpu.vram_mb(total capacity). With real free/used VRAM, it can route to workers that actually have headroom instead of the one with the biggest total card.What this is NOT: The full VRAM-accounted admission/eviction scheduler from #894. This is purely additive telemetry — no scheduler behavior changes.
Changes (6 files, +120 -1)
tinyagentos/cluster/worker_protocol.pyfree_vram_mb,used_vram_mb(default 0)tinyagentos/cluster/worker_capacity.pygpu_vram_snapshot()wrapssystem_stats.read_nvidia_vram()tinyagentos/worker/agent.pygpu_vram_snapshot(), sends VRAM in JSONtinyagentos/routes/cluster.pyHeartbeatBodyaccepts optional VRAM fields; endpoint forwards themtinyagentos/cluster/manager.pyheartbeat()stores VRAM when provided, skips when absent (legacy compat)tests/test_routes_cluster.pyBackwards compatibility
Tests
test_worker_registration_defaults_vram_to_zero— no VRAM sent → 0test_heartbeat_updates_vram— heartbeat with VRAM → stored + surfacedtest_vram_fields_survive_heartbeat_without_vram— legacy sparse heartbeat preserves prior valuestest_routes_cluster.pyunaffectedVerification
Summary by CodeRabbit
New Features
Bug Fixes
Summary by Gitar
/api/cluster/promote-archivedas a manual admin endpoint to trigger promotion across all online workers.tinyagentos/cluster/model_archive.pyto handle archive scanning, hardware requirement validation, and model file promotion.This will update automatically on new commits.