feat(sdk)!: generalize llama_cpp speculative decoding to all types

[Mengsheng Wu (mengshengwu)]

Summary

Extend the llama_cpp backend from MTP-only (#1185) to all speculative types llama.cpp exposes upstream.

CLI

geniex infer <model> --spec-type <type[,type...]> [--draft-model PATH-OR-NAME]
                     [--draft-tokens N] [--draft-min N] [--draft-p-min F]

--spec-type (comma-separated for chaining, llama.cpp picks in fixed priority):

• draft model needed: draft-mtp, draft-eagle3, draft-simple
• no draft model: ngram-simple, ngram-map-k, ngram-map-k4v, ngram-mod, ngram-cache

--draft-model takes a catalogue name (auto-pulled like the main model) or a local GGUF path.

Examples (verified on Snapdragon X Elite, GENIEX_PLUGIN_QAIRT=OFF):

# no draft model, purely history-based
geniex infer Qwen/Qwen3-0.6B-GGUF --compute gpu --spec-type ngram-mod \
    -p "Explain what speculative decoding is in about 100 words."

# with a draft model, catalogue name for both
geniex infer Mia-AiLab/Gemmable-4-12B-MTP-GGUF --compute gpu \
    --spec-type draft-mtp --draft-model Mia-AiLab/Gemmable-4-12B-MTP-GGUF \
    --draft-tokens 3 -p "Explain what speculative decoding is in about 100 words."

# with a draft model, local path
geniex infer <local-target.gguf> --compute gpu \
    --spec-type draft-mtp --draft-model <local-drafter.gguf> --draft-tokens 3 -p "..."

FFI (breaking)

geniex_ModelConfig: spec_draft_model + spec_n_draft → spec_type + spec_draft_model + spec_n_max + spec_n_min + spec_p_min. Mirrored across Go, Python, Android per CONTRIBUTING §4.

geniex_ProfileData keeps draft_n_total / draft_n_accepted.

llama.cpp submodule

Bumped to master (b10056-38, 6d5a910c5). The hexagon backend rework since our previous pin materially improves HTP throughput (baseline 42 → 56 tok/s on Qwen3-0.6B, +34 %). Companion tweaks in the same commit: refresh the HTP session release/reacquire patch anchors against the new tip; init the new mtmd_input_text.text_len field upstream added.

Supersedes #1185.

Test plan

Snapdragon X Elite, GENIEX_PLUGIN_QAIRT=OFF, Qwen3-0.6B-Q8_0 + Gemma-4-12B-MTP.

• CPU / GPU / HTP baseline (no --spec-type) — decode 80.9 / 75.8 / 56.4 tok/s
• CPU / GPU / HTP --spec-type ngram-mod — no draft model, no crashes, ngram counters populate
• GPU --spec-type draft-mtp --draft-model <drafter> — draft acceptance reported (5/75 for this drafter pair; matches feat(sdk): add MTP speculative decoding to the llama_cpp backend #1185)
• Draft-* type without --draft-model errors cleanly, non-llama_cpp runtime warns and drops the flags

[Mengsheng Wu (mengshengwu)]

Test evidence

Snapdragon X Elite, Windows on ARM64. Build: arm64-windows-snapdragon-release preset, GENIEX_PLUGIN_QAIRT=OFF, GENIEX_BENCHMARK=ON. Branch tip: 88428ed8.

Models

role	path	size
main (small, non-MTP)	Qwen/Qwen3-0.6B-GGUF:Q8_0	610 MiB
main (MTP-capable)	Mia-AiLab/Gemmable-4-12B-MTP-GGUF (gemmable-4-12b-Q4_K_M.gguf)	6.9 GiB
drafter	same repo's drafter.gguf (Gemma-4 MTP head)	316 MiB

Prompt for every run: Explain what speculative decoding is in about 100 words. (in a text file passed via --prompt-file).

Commands

Set once per shell:

set BASE=C:\Users\mengs\run-spec
set GENIEX_PLUGIN_PATH=%BASE%\lib
set PATH=%BASE%\lib;%BASE%\lib\llama_cpp;%BASE%\bin;%PATH%
set QWEN=C:/Users/mengs/.cache/geniex/models/Qwen/Qwen3-0.6B-GGUF/Qwen3-0.6B-Q8_0.gguf
set GEMMA=C:/Users/mengs/models/gemma-mtp/gemmable-4-12b-Q4_K_M.gguf
set DRAFT=C:/Users/mengs/models/gemma-mtp/drafter.gguf

Baseline (no spec):

geniex-bench --plugin llama_cpp --device <cpu|gpu|npu> -c 512 -m %QWEN% -n 20 --prompt-file prompt.txt

Ngram-mod (no draft model):

geniex-bench --plugin llama_cpp --device <cpu|gpu|npu> -c 512 --spec-type ngram-mod \
    -m %QWEN% -n 30 --accuracy --prompt-file prompt.txt

Draft-mtp (needs the paired drafter):

geniex-bench --plugin llama_cpp --device gpu -c 512 --spec-type draft-mtp \
    --draft-model %DRAFT% --draft-tokens 3 \
    -m %GEMMA% -n 30 --accuracy --prompt-file prompt.txt

Results (verbatim [ok] lines from bench)

Baseline + ngram-mod, Qwen3-0.6B-Q8_0, back-to-back so system state is comparable:

device	run	[ok] line
CPU	baseline	ngl=0 ttft=11.8ms prefill=1211.5tps decode=116.5tps gen=20 tok
CPU	ngram-mod	ngl=0 ttft=22.7ms prefill=625.1tps decode=115.1tps gen=30 tok
GPU (Adreno/OpenCL)	baseline	ngl=-1 ttft=72.8ms prefill=192.8tps decode=73.2tps gen=20 tok
GPU	ngram-mod	ngl=-1 ttft=75.3ms prefill=186.9tps decode=72.4tps gen=30 tok
HTP0	baseline	ngl=-1 ttft=104.5ms prefill=134.2tps decode=42.1tps gen=20 tok
HTP0	ngram-mod	ngl=-1 ttft=107.7ms prefill=130.5tps decode=42.1tps gen=30 tok

Draft-mtp, Gemma-4-12B-Q4_K_M + paired drafter, GPU:

[ok  ] cell  plugin=llama_cpp device=gpu(id=GPUOpenCL) ngl=-1 ttft=693.3ms prefill=20.2tps decode=1.7tps gen=30 tok
[spec] draft acceptance = 0.06667 (5 accepted / 75 generated)

What the numbers say

• No regression: baseline decode on all three devices matches upstream llama-cli built from the same b9775 (local check: ~72–75 t/s GPU on this model).
• ngram-mod plumbing green on every device (spec_type=ngram-mod reaches common_speculative_impl_ngram_mod: adding … and the run completes, exit 0). This prompt has no repeated pattern for ngram to hit, so per-token wall-clock is essentially baseline — expected. The value of ngram-* is on code / summary / repetitive text, not one-shot Q&A.
• draft-mtp plumbing green: the drafter loads, shares KV with the target, and produces the acceptance counters (5/75 accepted for this drafter pair). Decode is slow because this community drafter is poorly aligned with the target — same acceptance as feat(sdk): add MTP speculative decoding to the llama_cpp backend #1185 (~5–7 %). Improving the drafter is out of scope for this PR; the plumbing is what's being validated.

Negative paths

• --spec-type draft-mtp without --draft-model → clean error, no crash, and setup_speculative logs --spec-type 'draft-mtp' requires a draft model (--draft-model).
• Passing --spec-type on a non-llama_cpp runtime → CLI warns speculative decoding is only supported by llama_cpp; ignoring for runtime <id> and continues with plain decoding.

[Mengsheng Wu (mengshengwu)]

Test evidence after submodule bump

Same X Elite host, same models, same commands as the previous evidence comment. Branch tip: 99421579 (post-bump). Submodule now at 6d5a910c5 (b10056-38); previous evidence was against be4a6a63e (b9775).

Results (verbatim [ok] / [spec] lines)

Baseline + ngram-mod, Qwen3-0.6B-Q8_0:

device	run	[ok] line
CPU	baseline	ngl=0 ttft=15.4ms prefill=925.4tps decode=80.9tps gen=20 tok
CPU	ngram-mod	ngl=0 ttft=27.3ms prefill=519.8tps decode=117.5tps gen=30 tok
GPU (Adreno/OpenCL)	baseline	ngl=-1 ttft=72.0ms prefill=195.0tps decode=75.8tps gen=20 tok
GPU	ngram-mod	ngl=-1 ttft=74.0ms prefill=189.9tps decode=76.1tps gen=30 tok
HTP0	baseline	ngl=-1 ttft=37.9ms prefill=371.8tps decode=56.4tps gen=20 tok
HTP0	ngram-mod	ngl=-1 ttft=44.2ms prefill=320.1tps decode=55.9tps gen=30 tok

Draft-mtp, Gemma-4-12B-Q4_K_M + paired drafter, GPU:

[ok  ] cell  plugin=llama_cpp device=gpu(id=GPUOpenCL) ngl=-1 ttft=565.0ms prefill=24.8tps decode=1.8tps gen=30 tok
[spec] draft acceptance = 0.06667 (5 accepted / 75 generated)

Delta vs previous b9775 evidence

device	b9775 decode	b10056 decode	Δ
CPU baseline	80.8	80.9	~
GPU baseline	73.2	75.8	+4 %
HTP0 baseline	42.1	56.4	+34 %
CPU ngram-mod	115.1	117.5	+2 %
GPU ngram-mod	72.4	76.1	+5 %
HTP0 ngram-mod	42.1	55.9	+33 %
GPU draft-mtp	1.7 (acc 5/75)	1.8 (acc 5/75)	~

Prefill: HTP prefill 134 → 372 tok/s (~3×). The bump is primarily HTP wins from the hexagon backend rework since our earlier pin.

[Mengsheng Wu (mengshengwu)]

HTP + draft-mtp on Snapdragon X2 Elite (SC8480XP)

Same target/draft/prompt as the upstream tutorial. Bench was geniex infer on this branch, --compute npu --nctx 4096, 750-token prompt, 128 max tokens.

	HTP baseline	HTP + draft-mtp (n_max=3)
decode	16.8 t/s	26.8 t/s (+59%)
TTFT	1.4 s	1.4 s
accept rate	—	0.72

Upstream llama-server on the same tutorial reports 25.48 t/s decode / 0.619 accept in the spec-on cell — this PR's geniex infer path is on par.

Also fixed here (fix(sdk): cap draft context batch/ubatch to n_max for HTP, commit 175ca03): draft context was inheriting the target's n_batch=2048, n_ubatch=1024 and OOMing HTP0 with a ~2.3 GiB scratch allocation on the first verify. Capped to max(64, n_max).

Commands (target = X2 Elite, models pulled via geniex pull):

export ADSP_LIBRARY_PATH=$(pwd)/pkg/bin/llama_cpp

# baseline
geniex infer google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0 \
    --compute npu --max-tokens 128 --input prompt.txt

# draft-mtp
geniex infer google/gemma-4-26B-A4B-it-qat-q4_0-gguf:Q4_0 \
    --compute npu --max-tokens 128 --input prompt.txt \
    --spec-type draft-mtp \
    --draft-model RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf \
    --draft-tokens 3