Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ The first SwiftLM numbers from a **32 GB** Mac. Every other table in this README
| ~39.7K † | 757 / 31.0 | 53.3 s | 17.9 GB · 0 |
| ~80.7K † | 622 / 24.3 | 131.3 s | 19.5 GB · 0 |

Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6 (earlier build): 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). ⚠️ `--mtp` also currently changes the output at temperature 0: once the context passes Gemma's 1,024-token sliding window, rejected drafts aren't rolled back ([#184](https://github.com/SharpAI/SwiftLM/issues/184)). Avoid it until that's fixed.
Every needle check passed. `--mtp` with the bf16 assistant (`gemma-4-26B-A4B-it-assistant-bf16`) works but is slower on the M6 (earlier build): 45.2 / 35.6 / 30.4 tok/s decode at ~530 / 2.3K / 9.5K tokens, against 53.0 / 50.8 / 46.0 without it. A 4-bit MoE is compute-bound, so verifying the drafted tokens costs more than it saves (the same finding as the M5 Pro tables below). Rejected drafts are now rolled back exactly after the context passes Gemma's 1,024-token sliding window ([#184](https://github.com/SharpAI/SwiftLM/issues/184), [SharpAI/mlx-swift-lm#72](https://github.com/SharpAI/mlx-swift-lm/pull/72)). In a test on the M6, replacing every rejected draft with a different token left the output bit-identical. Output at temperature 0 can still differ from running without `--mtp`. The verify pass computes several tokens in one batch, so bf16 rounding differs from one-token decoding and can flip near-ties (`--mtp` also runs Gemma 4 through the text-only model code, while the plain model loads as a vision model). In our M5 Pro test prompts, every token `--mtp` picked was within 0.875 logits of the plain model's top choice with the bf16 assistant, and within 1.75 with the QAT 4-bit assistant.

### Qwen3.6-35B-A3B 4-bit — GPU vs SSD streaming

Expand Down
2 changes: 1 addition & 1 deletion mlx-swift-lm
Submodule mlx-swift-lm updated 22 files
+5 −4 .github/workflows/ci.yml
+3 −3 ...sting/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/GenerableRoundTripTests.swift
+3 −3 ...g/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/Grammar/EmitStopSignalTests.swift
+2 −2 ...ests/MLXFoundationModelsIntegration/GuidedGeneration/Grammar/FastForwardTokenizationDisagreementTests.swift
+7 −7 ...ntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/GuidedGenerationBenchmarkTests.swift
+6 −6 ...Testing/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/GuidedGenerationTests.swift
+3 −3 ...esting/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/HardReserveStressTests.swift
+6 −6 ...sting/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/MaxTokenTruncationTests.swift
+8 −8 ...tegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/MultiModelGuidedGenerationTests.swift
+2 −2 ...onTesting/IntegrationTestingTests/MLXFoundationModelsIntegration/GuidedGeneration/PrewarmGrammarTests.swift
+221 −101 Libraries/MLXFoundationModels/MLXDownloadProgress.swift
+43 −297 Libraries/MLXFoundationModels/MLXLanguageModel.swift
+298 −0 Libraries/MLXFoundationModels/ModelCache.swift
+40 −25 Libraries/MLXLLM/Models/Gemma4Text.swift
+144 −18 Libraries/MLXLMCommon/Evaluate.swift
+57 −6 Libraries/MLXLMCommon/KVCache.swift
+15 −0 Libraries/MLXLMCommon/KVCachePlan.swift
+405 −0 Tests/MLXFoundationModelsTests/MLXDownloadProgressTests.swift
+104 −0 Tests/MLXFoundationModelsTests/ModelCacheEvictionTests.swift
+123 −4 Tests/MLXLMTests/Gemma4Tests.swift
+366 −2 Tests/MLXLMTests/MTPSpeculativeDecodingTests.swift
+61 −0 Tests/MLXLMTests/MambaCacheCheckpointTests.swift
Loading