MatMulNBits / Q4_0: int4-compute CPU kernel — W4A8 (int8-dot) or W4A16 (f16-unpack)?

[czoli1976]

Follow-up to #2340 (int4 MatMulNBits → fused Q4_0 block-quant matmul) — a direction
question before I build the next piece. @kali

Where #2340 leaves things

#2340 keeps the weight as a Q4_0 constant instead of expanding it to f32 at load. What each
backend then does differs:

• Metal: the Q4_0 weight goes straight to the existing ggml mul_mv_q4_0 kernel — true
  int4 compute (locally ~2.3× vs f32 at K=N=4096, M=1).
• CPU: the block-quant packer dequantizes Q4_0 → f32 and runs the f32 microkernel. So CPU
  gets the memory win (~7× smaller weights) but only a bandwidth-grade speedup (~1.1–1.33×),
  not int4 compute.

So there's clear CPU headroom. Since int4 isn't a compute type, a "true int4 CPU kernel" means
unpacking to int8 or f16 and riding an existing GEMM — and tract already has both. Two paths,
with a numerical-contract tradeoff:

Option A — int8-dot (W4A8)

Unpack int4 → int8, quantize activations → int8, run the existing int8 GEMM kernels
(sdot #2278 / SME smopa #2279 / Intel AMX+VNNI #2339 / apple_amx).

• Fastest — rides all the int8 paths already in tree.
• But: activations become int8 too → effectively W4A8, a different contract than onnx: run MatMulNBits int4 through the fused Q4_0 block-quant matmul #2340
  (this is the llama.cpp Q4_0 × Q8_0 approach; near-lossless in practice, but it is a change).

Option B — f16-unpack (W4A16)

Unpack int4 → f16, use the existing f16 FMA kernels (fmla on NEON-fp16 / SVE).

• Keeps onnx: run MatMulNBits int4 through the fused Q4_0 block-quant matmul #2340's exact contract (f16/f32 activations).
• Faster than the current f32 dequant, slower than int8-dot. No new ISA needed.

The shared part

Either way, the only new code is ISA-agnostic: a Q4_0 → {int8 | f16} panel extractor (a variant
of extract_panel), plus — for A — on-the-fly activation quant + dual-scale combine. It reuses the
per-arch kernels already in tree; the one missing ISA piece is NEON smmla (I8MM), which would be a
bonus, not a prerequisite.

Question

For the CPU int4 path, which numerical contract do you want — A (W4A8, max speed) or
B (W4A16, preserve precision)? Or both — e.g. B as the safe default and A as an opt-in fast
path? Happy to build whichever you prefer.

[czoli1976]

One data-point that leans toward A (W4A8): tract's existing GPU backends already use it.

• CUDA routes Q4_0 weights to a GGML vec_dot_q4_0_q8_1 kernel and quantizes activations to
  Q8_1 first (cuda/src/ops/gemm.rs checks as_quant_fact(_, &Q4_0) then expects
  GgmlQuantQ81Fact activations; kernel in cuda/src/kernels/cu/mm_mv_q.cu). That's W4A8.
• Metal routes the same Q4_0 weight to its GGML mul_mv_q4_0 kernel.

So W4A8 (Q4_0 × Q8) is already the established, accepted-as-near-lossless path on GPU.

Correction to my own scoping (I checked the CPU side more carefully): a CPU W4A8 kernel
cannot just reuse the existing int8 GEMM. That GEMM applies a single output scale
(QScale in frame/mmm/fuse.rs), but Q4_0 carries a per-K-block scale (one per 32 elements),
and there's no Q4_0×int8 path on CPU today — only the f32-dequant extract_panel. So this needs a
dedicated fused int4×int8 microkernel — unpack each Q4_0 block → int8, quantize the matching
activation block → int8, int8-dot, scale per block, accumulate — i.e. ggml's
vec_dot_q4_0_q8_0, with per-ISA SIMD. The int8-dot instructions it would use are the ones
already in tree (sdot #2278 / SME #2279 / Intel AMX+VNNI #2339), but the kernel itself is new,
not glue.

Still leaning A (W4A8) to match the GPU path — I'd start with the decode (GEMV / M=1) case where
the int4 bandwidth win matters most. Happy to change course if you'd rather keep W4A16.

[czoli1976]

I prototyped the W4A8 CPU path locally to get real numbers for this decision (decode / GEMV only
so far). Summary:

Quality — W4A8 is ≈ W4A16: quantizing the activation to int8 (Q8) adds only ~0.3% on top of
the int4-weight error (which dominates at ~3.9% vs full f32). So Option A doesn't meaningfully cost
accuracy vs B.

Speed — at a 4096×4096 decode shape (M=1), the W4A8 path is ~1.80× faster than tract's own
mmv_f32 (0.81 ms vs 1.46 ms). That includes per-call op overhead in the prototype, so a real
kernel-selection integration would be ≥ that. It's in the same ballpark as the Metal mul_mv_q4_0
win, which fits.

Cost to implement — the pleasant surprise: the int8 dot autovectorizes to NEON sdot on its
own (integer reductions are associative, unlike strict-IEEE f32), so a correct and fast kernel
is ~50 lines of safe Rust — no per-ISA intrinsics, no unsafe. The dedicated fused kernel I
mentioned is still needed (per-K-block scaling doesn't fit the generic int8 GEMM), but it's far
cheaper than I'd feared.

So the data leans A (W4A8): near-lossless, ~1.8× decode, cheap. What I have locally is a standalone
Q4_0::w4a8_gemv + a demonstration op (validated end-to-end). The open question is the one piece
that's really yours: how it should integrate — a dedicated op routed by a transform (like the
Metal/CUDA Q4_0×Q8 ops), or via the MMM kernel-selection in strategize. Happy to turn it into a
PR in whichever shape you prefer.