From 41e5ce4fda7d8222157e6200c52b535141d1b548 Mon Sep 17 00:00:00 2001 From: Simba Zhang Date: Mon, 28 Sep 2026 08:42:12 -0700 Subject: [PATCH] chore: bump mlx-swift-lm to 9d5041a for the MTP rollback fixes Picks up SharpAI/mlx-swift-lm#72: rejected MTP drafts are rolled back layer by layer, including sliding-window layers after the ring wraps (#184). The range also has SharpAI/mlx-swift-lm#68 (upstream #603 sync, not used by SwiftLM). README: replace the "avoid --mtp" warning with the measured result. Rollback is exact; the remaining differences from plain decoding are batch-shape bf16 numerics and the text-only vs vision model path. Fixes #184 Co-Authored-By: Claude Sonnet 5.5 --- README.md | 2 +- mlx-swift-lm | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index fecbaf84..5f07b40e 100644 --- a/README.md +++ b/README.md @@ -86,7 +86,7 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README | ~39.7K | 757 / 31.0 | 53.3 s | 17.9 GB · 0 | | ~80.7K | 622 / 24.3 | 131.3 s | 19.5 GB · 0 | -Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). ⚠️ `--mtp` also currently changes the output at temperature 0: once the context passes Gemma's 1,024-token sliding window, rejected drafts aren't rolled back ([#184](https://github.com/SharpAI/SwiftLM/issues/184)). Avoid it until that's fixed. +Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6: 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). Rejected drafts are now rolled back exactly after the context passes Gemma's 1,024-token sliding window ([#184](https://github.com/SharpAI/SwiftLM/issues/184), [SharpAI/mlx-swift-lm#72](https://github.com/SharpAI/mlx-swift-lm/pull/72)). In a test on the M6, replacing every rejected draft with a different token left the output bit-identical. Output at temperature 0 can still differ from running without `--mtp`. The verify pass computes several tokens in one batch, so bf16 rounding differs from one-token decoding and can flip near-ties (`--mtp` also runs Gemma 4 through the text-only model code, while the plain model loads as a vision model). In our M5 Pro test prompts, every token `--mtp` picked was within 0.875 logits of the plain model's top choice with the bf16 assistant, and within 1.75 with the QAT 4-bit assistant. ### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming diff --git a/mlx-swift-lm b/mlx-swift-lm index 5596071b..9d5041a9 160000 --- a/mlx-swift-lm +++ b/mlx-swift-lm @@ -1 +1 @@ -Subproject commit 5596071bfe613abd35c2478fbe71ae03f10cb04f +Subproject commit 9d5041a9668d4533d408983f09f0364087a356a1