One folder per capability, one page each. Start with Getting Started — every other page builds on it.
| Capability | What it does |
|---|---|
| Getting Started | Initialize the runtime, load a GGUF model, build a context, and tokenize — plus the Arena/Freeable resource model. |
| Text Generation & Sampling | Stream a conversation token-by-token via ConversationState + DefaultLlamaIterator, controlled by a LlamaSampler chain, max tokens, and stop strings. |
| Chat Templates | Render role-tagged messages into a model-specific prompt with LlamaTemplate and the Role enum. |
| Log Probabilities | Collect per-token logprobs (sampled token + top-N alternatives) via setTopLogprobs. |
| Capability | What it does |
|---|---|
| Parallel Conversations | Decode many conversations through one shared context with BatchIterator, isolated by sequence id. |
| Speculative Decoding | Draft→verify→accept speedup via a draft model or n-gram prompt-lookup; greedy (lossless) or sampling (exact). |
| Distributed Inference (RPC) | Offload layers and KV-cache to remote rpc-server backends across machines. |
| Quantized KV Cache | Shrink KV-cache memory by quantizing the K/V cache ggml_type (flash attention for a quantized V cache). |
| Capability | What it does |
|---|---|
| Embeddings | Turn text into dense vectors with LlamaEmbedder over an embedding-mode context. |
| Reranking | Score documents against a query with LlamaReranker (BERT cross-encoders and Qwen3-style rerankers). |
| Decision Models | Detect native decision models and read one raw score per option with LlamaDecider (openjev, lev, nimble, kev, laya, clef). |
| Multimodal (Vision & Audio) | Attach images and audio to a prompt via MtmdContext (mmproj/clip projector). |
| Capability | What it does |
|---|---|
| Reasoning & Tool Calls | Tag reasoning / answer / tool-call sections during generation for per-section token counts and TOOL_CALL detection. |
| LoRA Adapters | Load a GGUF LoRA adapter and attach it to a model with initLoraAdapter. |
| Capability | What it does |
|---|---|
| Devices & Memory | Enumerate backends/devices, query CPU/GPU/RPC memory, read model dims, and read performance metrics. |
| Logging | Install a custom log callback over llama.cpp/ggml and filter by level. |
| Custom Builds & Platform Support | Build llama.cpp and regenerate the jextract FFM bindings for other platforms (Windows, ARM, CUDA, …). |