Support llama.cpp as a managed local runtime with GGUF downloads and a compute-backend selector

[YellowSnnowmann]

Summary

Support llama.cpp as a first-class local runtime: a bundled llama-server, a GGUF model downloader, and a compute-backend selector (CUDA / Metal / Vulkan / CPU) — the way OpenHuman already manages Ollama.

Problem

How to see the gap today

1. Open Connections → LLM (left nav, API keys group) → LLM Providers tab → Add a provider.
2. The Local runtimes group offers LM Studio, Ollama and OMLX (app/src/components/settings/panels/ai/ProviderAuthSection.tsx:51, labels at aiPanelTypes.ts:172-176). There is no llama.cpp entry, no GGUF browser, and no compute-backend control anywhere in the app.
3. The only way to reach llama.cpp is to build/install it yourself, run llama-server on 127.0.0.1:8080, and point OpenHuman at it as a generic local-openai: endpoint via config.toml / LOCAL_OPENAI_URL.

Today a llama.cpp user has to do all of the setup themselves and OpenHuman only meets them at the end of it.

• The managed local runtime — the one OpenHuman installs, spawns and supervises — knows exactly two providers: LocalAiProvider::{Ollama, LmStudio} (src/openhuman/inference/local/provider.rs:6-9). OMLX, MLX and local-openai are reachable as provider profiles (src/openhuman/inference/local/profile.rs) but the user runs the server themselves; nothing installs or manages them.
• llama.cpp is reachable only as a generic OpenAI-compatible endpoint: "llamacpp" | "llama.cpp" | "vllm" all map to LocalProviderKind::LocalOpenai (src/openhuman/inference/local/profile.rs:57), routed as local-openai:<model> against http://127.0.0.1:8080/v1 (profile.rs:185-201). The user must build or install llama.cpp, pick a backend at compile time, fetch the GGUF, and run llama-server before OpenHuman is any help. That path also carries ToolSupport::PromptGuided rather than native tool calling.
• There is no GGUF handling anywhere. grep -rni gguf over src/ and app/src returns two hits, neither a download path: an extension check in src/openhuman/inference/model_ids.rs:175 and a comment about Ollama's advertised context window in src/openhuman/inference/local/ollama.rs:374.
• There is no compute-backend concept — nothing chooses or reports CUDA vs Metal vs Vulkan vs CPU, and detect_gpu is best-effort enough that it reports "unknown" on non-Apple platforms (src/openhuman/inference/device.rs:28-33).

So the request is for llama.cpp to get the treatment Ollama already has: install it, download weights into it, pick the accelerator, and route to it — rather than being the fallback for "some OpenAI-shaped thing is listening on 8080".

Solution (optional)

• Add a LlamaCpp variant to LocalAiProvider with its own bootstrap: fetch a prebuilt llama-server for the host's platform and accelerator, spawn and supervise it the way the Ollama service path does (src/openhuman/inference/local/service/).
• A GGUF model manager: browse/download a quantisation into the workspace, and list what is already present, alongside the existing local-model picker.
• A compute-backend setting (Automatic / CUDA / Metal / Vulkan / CPU) that selects the matching llama-server build, with the resolved backend shown in settings.
• Give it a real provider profile (llamacpp:<model>) rather than folding it into local-openai, so its tool-calling and context-window behaviour can be described accurately instead of inheriting the generic defaults.

Note on scope: the same report also asked for keyless SearXNG web search, GPU/VRAM monitoring, and a computer-use button in the composer. The first two are tracked separately (see Related); they are not part of this issue.

Acceptance criteria

• Managed runtime — llama.cpp installs, starts, stops and reports status through OpenHuman, without the user running llama-server by hand.
• GGUF management — a model can be downloaded and selected from within the app, and already-present GGUFs are listed.
• Backend selection — the compute backend is selectable and the effective backend is displayed; an unavailable backend degrades to CPU with a visible reason rather than failing to start.
• Routing — llama.cpp is selectable per workload in AI routing with its own provider profile, not via local-openai:.
• Diff coverage ≥ 80% — the implementing PR meets the changed-lines coverage gate.

Related

• SearXNG is unreachable from chat: searxng_search is never registered as an agent tool #6127 — SearXNG is unreachable from chat (the keyless-web-search half of the same report)
• Add a live hardware monitor for CPU, RAM, disk and GPU/VRAM #6129 — Add a live hardware monitor for CPU, RAM, disk and GPU/VRAM (the GPU/VRAM half)
• src/openhuman/inference/local/provider.rs:6-9 (managed runtimes today)
• src/openhuman/inference/local/profile.rs:57, 185-201 (llama.cpp's current generic-endpoint handling)
• src/openhuman/inference/device.rs:28-33 (GPU detection is best-effort)

[tinysweeper]

Support llama.cpp as a managed local runtime with GGUF downloads and compute-backend selector, analogous to existing Ollama support.

Labelled priority: p2.

[YellowSnnowmann]

Feasibility

Verdict: large but well-precedented. The download/verify/unpack/cache machinery this needs already exists as a module and is already used in production for Node. Should be split into three.

The precedent that makes this tractable

tinyruntime (vendor/tinyruntime) is a TinyBus module whose entire job is: "resolve a language runtime, install one when the host has none, reuse one when it does" — its README describes the pipeline as download · verify · unpack · promote · reuse, with per-language backends (tinyruntime-nodejs, tinyruntime-python) behind a router that "knows nothing about any language". NodeBootstrap (src/openhuman/runtime/node/bootstrap.rs) is the host-side client for it, and runtime/javascript/mod.rs is the facade pattern for exposing it as a language slot.

That is exactly the shape a managed llama-server needs, and it means the hard, boring parts — archive integrity, atomic promotion into a cache, warm-start detection so a user is not re-downloading on every launch — do not have to be rewritten.

Second precedent: Ollama's managed lifecycle (src/openhuman/inference/local/install.rs, inference/local/service/) already covers install-and-supervise for an inference server specifically, including the awkward parts (orphaned-installer detection on Windows, spawn markers, admin/util paths).

What is genuinely new

1. Per-platform × per-accelerator llama-server builds. This is the real cost. CUDA / Metal / Vulkan / CPU × macOS / Windows / Linux, each a different artifact, each needing a source and a verification story. Unlike Node there is no single canonical distribution index to point at.
2. GGUF model management. No GGUF handling exists anywhere today — grep -rni gguf src/ app/src returns two hits, neither a download path (an extension check in inference/model_ids.rs:175 and a comment in local/ollama.rs:374). Browsing, downloading, quantisation choice, disk accounting, and listing what is already present are all new.
3. A real provider profile. llamacpp:<model> with its own LocalProviderProfile instead of inheriting LOCAL_OPENAI_PROFILE's generic defaults (inference/local/profile.rs:185-201) — notably ToolSupport::PromptGuided and default_context_window: None, both of which are wrong-by-default for a server whose GGUF advertises its own context length.
4. Promotion to a managed runtime. LocalAiProvider (inference/local/provider.rs:6-9) is the enum that gets install/spawn/lifecycle; adding a third variant pulls in the whole service surface around it.
5. UI. An entry in the Local runtimes group of Add-a-provider, a GGUF browser, and a compute-backend selector — none of which exist.

Dependency on #6129

The compute-backend selector is only meaningfully usable next to a VRAM readout — picking CUDA and a quantisation is guesswork without knowing how much VRAM is free. Worth sequencing the GPU half of #6129 before or alongside item 5 here.

Recommendation

Not a single PR. Split into:

• (a) managed llama-server runtime — a tinyruntime backend plus a LocalAiProvider::LlamaCpp variant, shipping CPU-only first so the pipeline is proven before the accelerator matrix.
• (b) GGUF model manager.
• (c) compute-backend selection + surfacing the effective backend.

(a) is the load-bearing piece; (b) and (c) are independently useful once it lands. Multi-week overall, but no step requires new architecture — which is the main thing worth recording here.