You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6f9f35b
Browse filesBrowse the repository at this point in the historyBrowse files
Every needle check passed in both modes. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below).
89
+
Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below).
90
90
91
91
### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming
92
92
@@ -97,16 +97,16 @@ Every needle check passed in both modes. `--mtp` with the bf16 assistant (`gemma
TurboKV barely changes speed on this model. Only 16 of its 64 layers use full attention (the other 48 are GatedDeltaNet), so the KV cache is already small. It saves about 1.5 GB at 40K tokens.
109
+
> **Correction:** an earlier version of these tables had `--turbo-kv` columns. Those runs passed `--ctx-size`, which on this mlx-swift-lm pin gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. They were really vanilla runs, so the columns were removed.
110
110
111
111
### What 32 GB exposed
112
112
@@ -121,7 +121,7 @@ TurboKV barely changes speed on this model. Only 16 of its 64 layers use full at
121
121
3.**An auto-detected VLM that failed to load exited the server.**`Qwen3.6-35B-A3B-UD-MLX-4bit` ships a `preprocessor_config.json` without `image_mean`. SwiftLM now falls back to text-only unless you pass `--vision`.
122
122
4.**Vision-capable models skipped chunked prefill.** On the older mlx-swift-lm pin, a text-only prompt on the VLM path ran through the model in a single pass. It's fixed by the mlx-swift-lm bump in #167. Every number in this section was measured on `main` with that bump.
123
123
124
-
> ⚠️ **`--turbo-kv`precision on M5 (not reproduced on M6):**on an Apple M5, Qwen3.8-27B-4bit with `--turbo-kv` gets exact long-range lookups wrong from somewhere between 2K and 5K prompt tokens. Asked how many numbered lines a prompt has, it answers "1,000"or "14" instead of 315 / 500 / 700. Without `--turbo-kv` it answers correctly, and on the M6 both modes are 24/24 correct from 2K to 11.8K tokens with the same prompt. The likely cause is a GPU-family-dependent path in TurboKV's dequant or attention kernels. Until it's fixed, avoid `--turbo-kv`on M5 when exact recall matters. Tracked in [#175](https://github.com/SharpAI/SwiftLM/issues/175).
124
+
> ⚠️ **`--turbo-kv`loses exact long-range recall on every chip:**once a prompt passes the 2,048-token compression threshold, Qwen3.8-27B-4bit with `--turbo-kv` gets exact lookups wrong. Asked how many numbered lines a prompt has, it answers "1,000", "1,314" or "14" instead of 315 / 500 / 700. Reproduced on both M5 and M6; without `--turbo-kv` it answers correctly. The cause is a cache-eviction bookkeeping regression: after compression, attention only sees the recent hot window, and positions restart. A fix is in progress; tracked in [#175](https://github.com/SharpAI/SwiftLM/issues/175). Until then, avoid `--turbo-kv` when exact recall matters. Also note that `--turbo-kv` currently has no effect when `--ctx-size` is set (the attention layers use `RotatingKVCache`).
125
125
>
126
126
> ⚠️ **Known issues:**`--gpu-layers N` (CPU/GPU layer partitioning) hits a Metal GPU timeout on the first request, on both M5 and M6 (repro: `--model mlx-community/gemma-4-26b-a4b-it-4bit --gpu-layers 23`). Tracked in [#176](https://github.com/SharpAI/SwiftLM/issues/176); the fix is [SharpAI/mlx-swift#17](https://github.com/SharpAI/mlx-swift/pull/17). Even once it's fixed, CPU-resident MoE layers are very slow (~0.4 tok/s prefill), so on a 32 GB Mac try `--stream-experts` first (Qwen3.6-35B-A3B: 13.2 tok/s decode, 7.7 GB GPU). QAT-quantized Gemma 4 MTP assistants (`…-qat-assistant-4bit`) fail with `unhandledKeys pre_projection/post_projection`; use `gemma-4-26B-A4B-it-assistant-bf16`.
Copy file name to clipboardExpand all lines: docs/profiling/m6/gemma4_26b_a4b_4bit.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,7 @@
1
1
### `mlx-community/gemma-4-26b-a4b-it-4bit`
2
2
3
+
> **Correction (Sep 24):** the `TurboKV` rows below were run with `--ctx-size`, which on mlx-swift-lm 460ff81 gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. So these rows are effectively vanilla, not TurboKV. See #175 for real TurboKV behaviour.
4
+
3
5
Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians
Copy file name to clipboardExpand all lines: docs/profiling/m6/qwen38_27b_4bit.md
+2Lines changed: 2 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,5 +1,7 @@
1
1
### `mlx-community/Qwen3.8-27B-4bit`
2
2
3
+
> **Correction (Sep 24):** the `TurboKV` rows below were run with `--ctx-size`, which on mlx-swift-lm 460ff81 gives the attention layers a `RotatingKVCache`, and `--turbo-kv` only applies to `KVCacheSimple`. So these rows are effectively vanilla, not TurboKV. See #175 for real TurboKV behaviour.
4
+
3
5
Apple M6 · 32 GB · runs=3 (long=1) · warmup=1 · gen=128 · temperature 0 · medians
0 commit comments