Skip to content

feat: multiple GPUs support and async dashboard spinner - #5

Open
gmandelli wants to merge 1 commit into
fabgoodvibes:devfrom
gmandelli:feat/multiple-gpus
Open

feat: multiple GPUs support and async dashboard spinner#5
gmandelli wants to merge 1 commit into
fabgoodvibes:devfrom
gmandelli:feat/multiple-gpus

Conversation

@gmandelli

Copy link
Copy Markdown

Summary

Owlrun previously assumed a single GPU end-to-end. Detectors only parsed the first nvidia-smi line / first WMI result, model fit math used a single VRAM number, and the Ollama subprocess
was launched with invalid CUDA_VISIBLE_DEVICES=all syntax and no multi-GPU scheduling hints. This PR makes multi-GPU rigs first-class on Linux, Windows, and macOS (eGPU).

The gateway registration payload is intentionally not modified.

Not Tested: Windows, Apple Silicon

Changes

Detection (internal/gpu)

  • New GPUDetail struct and Info.GPUs []GPUDetail. VRAMTotalMB is now documented as the SUM across cards.
  • New helpers Info.LargestSingleVRAMGB() and EffectiveVRAMGB(info, split).
  • New shared parseNvidiaSmi() (in nvidia_parse.go) that enumerates every GPU. Used by Linux, Windows, and macOS NVIDIA detectors.
  • Windows AMD detector switched to ConvertTo-Json -AsArray, dropped Select-Object -First 1, with a PowerShell 5 single-object fallback.
  • Apple Silicon path now populates GPUs[] with its single entry for consistency.

Ollama subprocess env (internal/inference)

  • New shared gpu_env.go builds platform-agnostic GPU env vars:
    • OLLAMA_NUM_GPU=99
    • OLLAMA_FLASH_ATTENTION=1 (NVIDIA) / 0 (AMD)
    • OLLAMA_SCHED_SPREAD=1 only when split is enabled AND Count > 1
    • Explicit device index lists (0,1,...,N-1) for CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES
  • Removed the invalid CUDA_VISIBLE_DEVICES=all literal.
  • ollamaEnv signature is now (info, gpuSplit) on all three platforms. Linux and macOS previously returned os.Environ() unchanged — they now actually emit GPU env vars.
  • Manager gained gpuSplit field, SetGPUSplit/GPUSplit accessors, passes the flag into ollamaEnv on subprocess start.

Config (internal/config)

  • New InferenceConfig.GPUSplit bool, loaded from [inference] gpu_split (default false).
  • New SaveGPUSplit(on) helper.

Dashboard (internal/dashboard)

  • Status.GPU gained Count, Split, and GPUs []GPUEntry for the per-card breakdown.
  • New SetGPUSplitFunc setter, /api/set-gpu-split route, handleSetGPUSplit.
  • New HTML toggle #gpu-split-row (hidden via JS unless gpu.count > 1), JS function setGPUSplit, and a status-sync hint that displays the actual largest-single vs pooled VRAM numbers.

Tray wiring (tray.go, tray_linux.go, tray_other.go)

  • Status fill populates Count, Split, and GPUs[].
  • All model-selection callers now use gpu.EffectiveVRAMGB(info, cfg.Inference.GPUSplit). Default behaviour is largest single GPU — safe upper bound for models that don't shard across
    cards.
  • After Manager creation, SetGPUSplit(cfg.Inference.GPUSplit) is invoked so the initial Ollama launch uses the right env.
  • New SetGPUSplitSetter saves config, updates the manager, restarts Ollama, re-selects models with the new effective VRAM, reloads the primary, and reconnects to the gateway. Rejects the
    call when Count < 2.

Installer (installer/install.sh, installer/install.ps1)

  • Both installers enumerate every GPU instead of stopping at the first.
  • Step 1 prints a per-GPU breakdown when Count > 1.

Behaviour changes for users

  • Single-GPU users: no functional change. The dashboard split toggle is hidden.
  • Multi-GPU users: model fit defaults to the largest single card (previously summed all VRAM, which was overly optimistic). A new dashboard toggle ("Split model across GPUs") opts into
    pooled VRAM with OLLAMA_SCHED_SPREAD=1 and triggers an Ollama restart + model reselection.

Async operation feedback

  • New server-published busy field on /api/status for long-running ops.
    handleSetGPUSplit and handleSwitchModel set it for the duration of the
    call.
  • Dashboard shows a fixed top banner ("Applying GPU split…", "Loading model X…")
    with a spinner and disables buttons/checkboxes while busy. Optimistic update
    on click; server state takes over on the next poll.

Model download visualization fix

  • Server now publishes pulling_model (the tag currently being downloaded)
    alongside pulling.
  • Dashboard model list previously fell through to a generic spinner branch for
    every model whenever any pull was active. Fixed: only the model matching
    pulling_model shows the "Downloading…" spinner. Other installed models keep
    their Activate/Remove buttons; non-installed models render a disabled
    Download button until the in-flight pull finishes.

What is NOT changed

  • internal/marketplace/gateway.go and registerPayload are untouched (per request).

Test plan

  • make build-all (Linux, Windows, macOS amd64/arm64) passed
  • go test ./internal/gpu/... ./internal/inference/... ./internal/dashboard/... ./internal/config/... passed
  • make test (full suite) passed
  • Manual: 2x NVIDIA rig — verify dashboard shows both cards, toggle restarts Ollama, OLLAMA_SCHED_SPREAD=1 is set when on passed
  • Manual: 2x AMD rig on Windows — verify WMI enumeration returns both cards not tested
  • Manual: single-GPU rig — verify toggle row stays hidden and behaviour is unchanged passed
  • Manual: Apple Silicon — verify no regressions not tested

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant