🔄 Auto-Sync: Apple Upstream Repository - #62
Open
solderzzc wants to merge 235 commits into
Open
Conversation
BERT models crash when input exceeds maxPositionEmbeddings because the position embedding table is fixed-size. Truncate with a warning rather than crashing. Expose maxPositionEmbeddings on the EmbeddingModel protocol (default nil, non-breaking) so callers can check the limit and pre-truncate or chunk as needed. Fixes #62.
* Bump swift-syntax dependency * Update and fix usage examples
ml-explore#149) Qwen35Language.LanguageModel.callAsFunction assumes inputs is always 2D [batch, seq], but text-only callers like WiredMemoryUtils.tune and TokenIterator can pass 1D [seq] token arrays. This causes getRopeIndex() and subsequent dim(1) calls to crash with "SmallVector out of range" when accessing a non-existent dimension. Add an ndim check at the top of callAsFunction to expand 1D inputs to 2D before any dimension-dependent logic runs. Fixes ml-explore#148
* Add coherence integration tests * Consolidate task registration
…xplore#168) (ml-explore#170) `TokenRing.loadPrompt` used `prompt.dim(0)` to count tokens, which returns 1 for VLM models that pass [1, n]-shaped prompts. This caused the ring buffer to be incorrectly sized, leading to a broadcast shape crash on the next `append` call during generation. Flatten the prompt to 1D upfront so the token count and all downstream slicing work correctly regardless of input shape. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…lore#167) The tools schema was passed to the chat template (so the model knew about the tools) but never forwarded to the ToolCallProcessor that parses the response. Without it, array and other non-string parameter types in XML tool call formats (Qwen 3.5) were returned as raw strings instead of being decoded to their proper types. Fixes ml-explore#159.
…ml-explore#174) GlmOcr never forwarded tools or additionalContext to applyChatTemplate (added after ml-explore#140, missed the pattern). SmolVLM2's video path was also missing them while its other two branches had them.
* fix: use preconcurrency CoreImage import
…ore#164) feat: add ParoQuant model support Load PARO-quantized models (AutoAWQ format) with pairwise Givens rotation applied at runtime via a Metal kernel. Key additions: - RotateQuantizedLinear: Metal kernel for pairwise Givens rotation + quantized matmul. Rotation state derived once at load time (thread-safe under concurrent inference). - ParoQuantLoader: AutoAWQ→MLX weight conversion, rotation layer patching, fused in_proj_ba splitting for Mamba projections. - maybeQuantizeKVCache fix: handle hybrid caches (attention + Mamba) where cache[0] isn't a KV attention cache. - Unit tests covering pair packing, AWQ conversion, quantization round-trip, and concurrent safety.
* Handle stringified JSON tool call arguments
* fix build
…l-explore#229) MLXArray(-Float.infinity) creates a float32 scalar. which() then promotes the entire bf16 segsum output to fp32, doubling memory per SSM layer. At L=2048 this wastes ~960MB on Qwen3.5-35B (30 GDN layers, 4 heads) and ~24GB on Nemotron-30B (48 Mamba layers, 64 heads) per prefill. Fix: match the -inf scalar to the accumulator dtype.
…-explore#193) * feat: expose speculative decoding in ChatSession (ml-explore#181) * feat: add generateTask overload for SpeculativeTokenIterator Add SpeculativeDecodingConfig struct and wire SpeculativeTokenIterator into ChatSession.streamMap, enabling ~2-3x generation speedup with no API break for existing callers. Exposes a public generateTask(iterator: consuming SpeculativeTokenIterator) overload so that ChatSession can obtain the (stream, task) pair needed for clean early-termination handling — matching the pattern used by the existing TokenIterator overload. --------- Co-authored-by: David Koski <46639364+davidkoski@users.noreply.github.com>
…ore#223) Fix EmbeddingGemma weight loading (`sanitize(weights:)`) Two bugs prevented loading any `mlx-community/embeddinggemma-*` checkpoint: 1. **Init-order crash** — `sanitize(weights:)` assigned the dense head via `self._dense.wrappedValue = ...`, which fatals when `module != nil`. Route through `update(modules:)` instead. 2. **Wrong hidden size** — the dense head was initialized with `config.intermediateSize` (the backbone MLP dim), but the actual checkpoint expands 4× to `hiddenSize * 4`. Read the dim directly from `dense.0.weight` shape instead. --- Co-authored-by: Anatoly Samoilenko anatoly.samoilenko@gmail.com Co-authored-by: David Koski dkoski@apple.com
…ustion (ml-explore#226) CIContext() default options cache IOSurface-backed GPU textures for intermediate filter results. Each frame in the VLM image pipeline (tone-curve → bicubic resample → color-matrix normalize) allocates multiple IOSurfaces that are held in the context cache across calls. Processing a large library (hundreds of videos, each yielding several frames) accumulates thousands of cached surfaces and hits the macOS per-process IOSurface kernel limit of 16384, causing render failures: IOSurface creation failed: e00002be (likely per client IOSurface limit of 16384 reached) -[CIContext _startTaskToRender:...] Render failed because of failure to allocate intermediate. Fix: pass `.cacheIntermediates: false` when constructing the shared context. Batch inference never re-renders the same CIImage twice, so the intermediate cache provides no benefit. The change eliminates the surface accumulation while leaving all functional behavior intact. Co-authored-by: Vladimir <vladimir@sinitcin.com>
…plore#228) Softmax was applied to all 128 expert scores before top-k selection, so selected weights came from the full distribution rather than just the top-k. Moved softmax to after selection so it operates only on the chosen experts. Drops the renormalization step since softmax on top-k already produces a valid distribution. Also fuses norm + scale from 3 dispatches into one MLXFast.rmsNorm call.
…l-explore#227) SwitchGLU variant for models that ship a single fused gate_up_proj weight of shape [numExperts, 2*hiddenDims, inputDims] instead of separate gate_proj / up_proj. Gemma 4 26B MoE uses this layout. One gatherMM dispatch for the combined projection, split, then activate. Same gatherSort optimization as SwitchGLU.
) (ml-explore#253) Fix UserInput init not populating self.images/self.videos (ml-explore#182) Property observers don't fire during initialization, so prompt.didSet — which keeps self.images/self.videos in sync — was never called by two of the inits: init(prompt: String, images:videos:tools:additionalContext:) built a .chat prompt from its parameters but never copied images/videos into self. init(prompt: Prompt, images:videos:...) had an explicit case .chat: break, silently dropping the parameters. Both inits now mirror init(chat:processing:tools:additionalContext:): extract images and videos from the chat messages, or use the explicit parameters for non-chat prompts. Add regression tests for UserInput init image/video sync (ml-explore#182) Three tests pinning the contract that self.images/self.videos are populated at construction, not deferred to the next prompt assignment: testInitFromPromptStringPopulatesImages — string-based init reflects the images parameter immediately. testInitFromPromptEnumPopulatesImagesForChat — the .chat branch of the Prompt-based init derives images from chat messages rather than dropping them. testInitFromPromptStringPopulatesVideos — symmetric coverage for the videos parameter.
…re#225) * pipeline prefill chunks with asyncEval -- 10x on GDN models eval(cache) between prefill chunks was a blocking sync -- CPU idle while GPU worked, then built next chunk serially. Python mlx-lm avoids this because eval is deferred until a value is read. asyncEval on cache state per chunk lets CPU build ahead while GPU pipelines through. One terminal eval(cache) after the loop. Prefill throughput (tok/s), Qwen3.6-35B-A3B-4bit, M5 Max 128GB: ctx | before | after | speedup 128 | 260 | 696 | 2.7x 512 | 235 | 2201 | 9.4x 1k | 244 | 3130 | 12.8x 2k | 270 | 3937 | 14.6x No regression on dense models (Gemma 4 E2B, Llama 3.2 3B). Decode unchanged -- different code path.
…lore#224) * fix gated delta state precision -- fp32 state to match Python State was typed as q.dtype (bf16), losing precision across T-step recurrence. Python mlx-lm keeps state in fp32. Aligns Swift with that. - kernel source: write o_state as StT not InT - dispatch: add StT template, use state.dtype for output - gatedDeltaUpdate: create state in fp32, upcast if needed Before: kernel showed ~0.25 max diff vs ops fallback at T>1. After: matches ops path. Tested on M5 Max 128GB, Qwen3.6-35B-A3B-4bit. Co-authored-by: tturney <tturney@psyguard.ai>
Removed the command to show SDK build version during the build process. This was informative, but ultimately causing problems with installations and not actually guarding anything.
* models should not mutate state during eval - fixes ml-explore#157 * sync gptoss with mlx-lm -- no missing sinks
) Fix Gemma4TextBackbone.callAsFunction crash on 1D token input When callers construct LMInput directly (e.g. for manual KV-cache reuse), tokens arrive as 1D (L,) instead of 2D (1, L). This causes processedPerLayerInputs to be 3D, and the 4D subscript on finalPerLayerInputs crashes in mlx_array_dim. The first request succeeds because autoregressive step adds .newAxis; the crash hits on the initial prefill of continuation requests. Fix: expand 1D inputs / 2D inputsEmbeds to add a leading batch dim at the top of callAsFunction. Zero-copy reshape; no behavior change for callers already passing the canonical 2D shape.
Fix Gemma4VisionPooler kernel derivation to use padded sequence length
Previously, the kernel was derived from the real patch count, which could yield kernel=2 at the 280-token budget instead of the expected 3, causing real patches to map to zero rows in the einsum output.
Extract gemma4VisionPoolingKernel(paddedPatchCount:outputLength:) and derive the kernel from pooledHiddenStates.dim(1) to consistently return pool=3 across all supported budgets. Add regression tests for all five budgets {70, 140, 280, 560, 1120}.
* Optimize Qwen MoE expert reduction * Enable Qwen direct expert reduction by default
* Fuse Qwen GDN input projections * Move inference preparation to LanguageModel lifecycle * Use prepare overload for inference lifecycle
…l-explore#594) * Stop calling deprecated MLX memory APIs from the integration tests MLX moved its memory inspection and cache control onto a Memory type and deprecated the older GPU spellings. The integration tests still used the old names, so every build of the IntegrationTesting scheme printed a dozen deprecation warnings that buried the warnings actually worth reading. Switch the call sites over to Memory.snapshot() and Memory.clearCache(), and fix a doc comment that pointed at the old name. The deprecated calls just forward to the new ones, so behavior is unchanged. * Move Gemma's video frame handling off the deprecated VideoFrame API Gemma was the last video model still building frames through the old VideoFrame spellings, which take and hand back a CoreImage image directly. Every other vision model already goes through the newer form that wraps the image in the general-purpose image case, so this was the only source of these deprecation warnings left in the library. Reading the incoming frame through the supported accessor also removes a force-try: the deprecated property traps when the frame is not backed by a CoreImage image, whereas the surrounding frame-processing closure is already allowed to throw, so the failure now surfaces to the caller instead of killing the process. * Remove a leftover unused local from Gemma's text backbone An earlier fix to Gemma's cross-layer key/value sharing stopped consulting whether the caller supplied a cache when deciding which layers reuse an earlier layer's keys and values. That fix removed both readers of the local variable holding that answer but left the variable itself behind, so the compiler has been reporting it as an unused value ever since. Drop the declaration. Its initializer only tested an optional for nil, so nothing observable changes. * Clear the remaining compiler warnings from the package With the deprecation warnings fixed, the build still reported a handful of style diagnostics. Most were `var` bindings that are only mutated through subscript assignment or a copying update, which `let` already permits, and `try`/`await` markers on expressions whose callees turned out to be synchronous. Breakpoints of note: the optional-key-path mock in the chat-session tests calls a synchronous prepare overload, and the reranker test used a conditional cast to an existential to check a protocol conformance that is statically guaranteed; the coercion form still fails the build if that conformance is ever removed, but no longer warns. The one warning needing care was ParoQuant's concurrency test, which captures a non-Sendable layer in the @sendable closure of DispatchQueue.concurrentPerform. Unsynchronized access is the whole point of that test, so the layer now crosses the boundary in a small unchecked-Sendable wrapper, alongside the existing thread-safe accumulator in the same file.
* Add the CI state label module with the run-start rule A pull request whose CI run has just started carries ci-running, and loses the labels that described the previous run. An approval by a maintainer survives the start, because the completion rule needs to see it. * Turn a finished CI run into one pull request label A green run asks for review, or reports the pull request ready to merge when a maintainer has already approved it. A red run names what the author must fix, and only calls for formatting work when formatting is the single thing that failed. A run that failed on the build image, or that was cancelled, asks for another run instead of blaming the author. * Put a pull request back in the CI queue when a new commit arrives Every CI result and every approval describes the commit that has just been replaced, so all of them go. The pull request is only queued again once a scan has read it, which its category label proves. * Read the run and write the labels through an injected client The three functions that talk to GitHub take the client as an argument, so the tests inject a fake and none of them reaches the network. Finding the pull request from the head commit is what makes this work for a fork, whose run payload may carry no pull request at all. * Choose the pull request the run actually tested, and test the requests A commit can belong to more than one open pull request when branches are stacked, so the head commit now decides which one the label describes. The fake client also records what each read asks GitHub for, so a wrong run number can no longer pass the tests and fail only in a live run. * Mark the build-image steps and run the label classifier's tests Three steps check the state of the build machine rather than the change, so their names now say so. A failure in one of them asks for another run instead of asking the author for a fix. A new job runs the classifier's own tests, so a change to it cannot merge untested. * Label a pull request while CI runs and when it finishes A pull request shows ci-running from the moment its run starts, and gets one label describing the result when the run ends. Both handlers ignore a run whose commit the pull request has already left, so nobody reviews code that has been replaced and no newer commit loses its place in the queue. * Requeue a pull request for CI when a new commit arrives Every label describing the previous commit goes, and the pull request waits for CI again. A pull request no scan has read is cleared but not queued, so new code never reaches the build machine before it has been read. * Ignore a CI run that never started, and stop pinning a stale base commit A run held for approval reports a conclusion without having run, so the completion handler now ignores it rather than clearing the labels a maintainer set. The reset workflow also takes the default branch instead of the pull request's base commit, because an older base does not yet contain the script the job loads. * Say what the reset workflow's checkout actually takes The comment still described a pinned base commit after the checkout changed to the default branch. It describes the security property of a job that holds a write token, so a stale version of it is worse than none. * Accept the old build-machine step names, and correct four comments A pull request from a fork runs its own copy of the CI workflow, so one branched before the step rename still reports the old names. The classifier now treats both spellings as a build-machine failure, rather than asking a contributor to fix a machine they cannot reach. Four comments also said more than was true: about which files a workflow loads, about what a checkout takes, about why one clause exists, and about what never cancelling guarantees. * Say when the legacy step names can go, without naming another repository The note pointed at a file a reader of this repository cannot see. It now states the condition on its own terms: the list can go once no open pull request predates the rename. * Copy edits * Read the CI workflow's contract number from a step name * Ask for another run when the rules do not match the branch * Let the contract number replace the list of old step names * Name the contract number in the CI workflow * Log the branch's contract number next to the expected one * Fix two tests that passed for the wrong reason Both tests check that a failure the author cannot fix gets the needs-ci label. Their sample job data left out the contract step. One test should reach the no-failed-step check, and the other should reach the build-machine check. Both reached the newer contract check instead. With the build-machine check deleted, every test still passed, so nothing guarded that check. This change adds the contract step to both samples. Each test then reaches needs-ci through its own check again. If you delete the build-machine check now, its test fails.
…ensors (ml-explore#598) An MLX-converted checkpoint may keep its mtp.* tensors for speculative decoding. Treating their presence as proof of a raw upstream checkpoint applies the zero-centered RMSNorm +1 shift a second time, leaving every layernorm near 2.0 instead of 1.0 and degrading generation to garbage tokens. The conv1d layout identifies raw checkpoints on its own: transformers exports store conv1d as [C, 1, K], and a converter that transposes conv1d shifts the norms in the same sanitize pass, so the two signals cannot diverge in a valid checkpoint. MTP-only drafter checkpoints are unaffected because the mtp. filter removes all their tensors before the shift loop.
…tchGLU (ml-explore#511) * Allow downstream specialization of Qwen3.5 GDN/MoE blocks and SwitchGLU `Qwen35Language` has no access modifier, so everything nested in it is internal regardless of its own modifier, and `GatedDeltaNet`, `SparseMoeBlock`, `DecoderLayer` and `Model` are additionally `final`. `SwitchGLU` and `SwitchLinear` are `public` rather than `open`. Together this makes the Qwen3.5 language stack impossible to specialize from another module. Serving this model with routed experts streamed from SSD instead of resident needs exactly one substitution — our own expert MLP in place of `SparseMoeBlock`'s — while keeping the rest of the vendored model and its weight keys intact. Without subclassing that becomes a reimplementation: ~445 lines re-deriving the `@ModuleInfo(key:)` bindings, the gather/scatter sort and the quantized-vs-dense dispatch that `SwitchGLU` already gets right, and every upstream fix has to be hand-ported. Widens access only: Qwen35Language -> public GatedDeltaNet, SparseMoeBlock, DecoderLayer, Model -> open SwitchGLU, SwitchLinear -> open SwitchGLU/SwitchLinear/SparseMoeBlock callAsFunction -> open `Module` is already `open`, so nothing changes in mlx-swift. `SparseMoeBlock.callAsFunction` had to widen as a consequence, not a choice: it satisfies a requirement in the public `UnaryLayer` protocol, so an `open` class cannot leave it internal. Made `open` rather than `public` since overriding it is the point. Leaves FusedGateUpSwitchGLU alone and omits the GDN conv-fold port, which is a behaviour change and belongs in its own PR. Full suite: 296 tests, 0 failures. * public/open updates --------- Co-authored-by: Fred Jura <digital@afya.ai> Co-authored-by: David Koski <dkoski@apple.com>
…e#596) Mirrors the Gemma 3/4 encoder exposures (ml-explore#387, and the Gemma 4 follow-up) for Falcon-H1, so client code can run the decoder stack over an embedding it built itself rather than over a single token id. The motivating case is a DualAR TTS model (Audio8 TTS Preview 0.1b, model_type `arktts`, `slow_backbone: falcon_h1`) whose slow-stack input is a text embedding summed with ten codec-codebook embeddings. Such a client never has one token id to look up, so `FalconH1ModelInner.callAsFunction(_:mask:cache:)` is unreachable for it, and today the only options are to fork the file or re-implement the mask setup and layer loop against internal details. Exposed at @_spi(FalconH1Encoder) scope, keeping this off the advertised public API of MLXLLM while making it usable outside the module: * FalconH1Configuration.hiddenSize / .numHiddenLayers / .rmsNormEps / .embeddingMultiplier * FalconH1DecoderLayer and its callAsFunction * FalconH1ModelInner.init, .embedTokens, .layers, .finalLayerNorm * FalconH1ModelInner.callAsFunction(inputsEmbeds:cache:) — new * FalconH1ModelInner.callAsFunction(_:mask:cache:) — was internal `callAsFunction(inputsEmbeds:cache:)` is the seam; the token-id overload now delegates to it, so the two paths cannot drift. No behaviour changes for existing callers: FalconH1Model still embeds, runs the stack and applies the head exactly as before. `FalconH1ModelInner.init` is exposed because constructing `FalconH1Model` is not an equivalent way in: the wrapper adds an `lm_head` whenever `tie_word_embeddings` is false, which is the decoded default. A client bringing its own head does not want that module and would otherwise have to account for it when matching module keys against a checkpoint. `FalconH1Model.model` and the mask helpers (`createSSMMask`, `createAttentionMask`) were already public, and none of the layer's stored property types are `private`, so unlike the Gemma 4 change nothing else had to widen. FalconH1EncoderAccessTests proves sufficiency the way the Gemma access tests do: it drives the stack using ONLY the exposed surface, importing MLXLLM without @testable and opting in via @_spi(FalconH1Encoder). It checks shape, determinism, that the seam agrees with the token path, that the layer stack can be driven a layer at a time to tap every hidden state, and that the stack actually transforms its input — the last so the suite cannot pass on a seam that returns its argument or runs zero layers. One test earns its place beyond access checking. `sanitize()` folds `embedding_multiplier` into `embed_tokens.weight`, which is right when the token lookup is the only thing entering the stack but is a live trap for a composite- embedding client: fold it and add unscaled contributions, and the lookup half is scaled while the rest is not. Measured against the PyTorch reference for the model above, that is 77% relative error on the embedding and 38% on the final hidden state — while still producing plausible audio of the right length, so listening does not catch it. testCompositeEmbeddingMultiplierEquivalence pins `(raw + other) * m == folded + other * m` and asserts the unscaled-remainder variant is observably different, both at the embedding and after the stack, so the wrong form cannot quietly become the expected one. Runs on a tiny randomly-initialized model; no weights are downloaded. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
- Restore the per-response usage update the adapter sends to FoundationModels. A July fix removed it because the beta SDK declared a call the shipping system library did not provide, and every request crashed the moment generation finished. - Confirm the mismatch is gone on Xcode 27 beta 6 with the macOS 27 and iOS 27 SDKs. The system library now provides the call, and a real request completes normally. - Before this change a caller read zero prompt tokens. Completion tokens were already counted from the streamed text, so totals looked plausible and the loss was easy to miss. - Add integration tests for the usage a caller actually reads: a new session starts at zero, one response reports both prompt and completion tokens, a session sums usage across turns, two sessions stay independent, and the streaming path agrees with its final snapshot. - Confirm the new tests fail without the restored update and pass with it. - Note the new tests need a macOS 27 host, so CI does not run them today.
…l-explore#579) * Suspend instead of blocking cooperative threads while weights load The Load API is synchronous and blocks its thread on file I/O and DispatchQueue.concurrentPerform, but every model factory calls it from an async context, parking a Swift concurrency cooperative thread for the whole load. Add an async loadWeights overload that hops to a global queue and suspends the caller, and use it from the factories. The synchronous overload remains for synchronous callers such as convert. * Disambiguate loadWeights doc links now that an async overload exists The new async loadWeights overload made three DocC symbol links ambiguous, failing the docs verification in CI. Point each link at the intended overload: the async one (-8b333) from porting.md, which describes the inference path, and the synchronous one (-6eqw7) from the model-conversion docs and the async overload's own doc comment.
- In `GenerationEventObserverTests`, two tasks drained the generation channel with an empty `catch {}` block. A read failure there would pass the test silently. Extracts one drain helper. Each test now awaits its result and accepts only a cancellation error.
- Asserts the expected error from every load and warmup task instead of discarding it with `try?`.
- The downloader and tokenizer-loader test stubs were classes with no stored state, marked `@unchecked Sendable`. That annotation tells the compiler to trust the type is safe to share across threads, without the compiler checking it. Converts the stubs to structs, which are `Sendable` by construction, so the compiler proves the guarantee instead of the annotation asserting it.
- `EventBox` and `SeedRecorder` hold mutable state a closure captures across a task boundary, so a struct will not work for them. They used `NSLock` with the same `@unchecked Sendable` escape hatch as the stubs above. Replaces the lock with the standard library's `Mutex`, so these types are provably `Sendable` too, instead of asserting it.
- In `GenerationEventObserverTests`, `withValue` took two closures, one passed as an argument and one as a trailing closure. swift-format's `OnlyOneTrailingClosureArgument` rule flags that form because it is easy to misread which parameter each closure binds to. Spells out the second closure with its argument label instead.
- `EvictBiasStubTokenizerLoader.load()` returns a `CountingTokenizer`, which requires OS 27. The old code checked that at runtime with a `guard` and a `preconditionFailure` that could never actually fire. Marks the stub itself with the OS 27 requirement, so the compiler enforces it and the dead runtime check is gone.
An empty catch and a discarded task result both hid a real loader failure, so a broken load could not fail a test. The fixtures also claimed thread safety by annotation where value semantics or a mutex states it, and asserted an availability requirement at runtime that the type system can enforce at compile time.
…follow-up) (ml-explore#471) feat(paroquant): MoE architecture support, prepared checkpoint caching, and rotation kernel optimizations ### MoE Support & Architecture - **PairwiseRotation & RotateSwitchGLU:** Extracted standalone `PairwiseRotation` module and implemented `RotateSwitchGLU` to support MoE PARO models (e.g., Qwen3.6-35B-A3B). - **MoE Loader Pipeline:** Added passes to stack per-expert AWQ weights, remap shared rotations, and patch switch GLU layers without mutating dense paths. - **Conversion Robustness:** Dropped theta-filter to convert un-partnered expert weights, match scales/biases to checkpoint float dtype, and optimized prefix matching from $O(N \cdot M)$ to $O(1)$. ### Performance - **Pre-Gather MoE Rotation:** Moved `gate_up` rotation before expert gather/sort, reducing row-rotation overhead by $1/\text{topK}$. - **Simdgroup Rotation Kernel:** Rewrote the Metal rotation kernel for `groupSize == 128` to execute in a single simdgroup with `simdgroup_barrier` (with generic fallback for other sizes), delivering up to 2x speedup. - **GatedDelta Fusion:** Compile-fused the 6-kernel elementwise decay gate chain into a single dispatch. - **Prepared Checkpoints:** Added background caching of converted weights to `prepared_checkpoint.safetensors` with self-healing validation manifests, cutting warm-load times significantly. ### Fixes & Cleanups - Unified `RotationDerivedState` and batched GPU evaluation across all rotation modules at load time. - Froze `PairwiseRotation` parameters to allow fused weighted reduction in `SwitchGLU`. - Fixed tool-call format and chat convention resolution in `loadParoQuantModel`. - Fixed volume free-space detection for disk persistence on CI runners. Refs ml-explore#208, ml-explore#209, ml-explore#220, ml-explore#424, ml-explore#471
…e#605) - Adds `await` to the two `loadWeights` calls in `Gemma4AssistantDraftModelIntegrationTests.swift`, both inside `async throws` test functions. - Adds `await` to the `loadWeights` call in `loadTargetAndDrafter` in `MTPIteratorEndToEndDiagnosticTests.swift`. - Adds `await` to the two `loadWeights` calls in `loadRung4Drafter` in `MTPRung4TokenParityTests.swift`. ml-explore#579 added an async `loadWeights` overload next to the existing synchronous one. Swift prefers the async overload inside an async function, so a plain `try loadWeights(...)` in an async function now needs `await`. These five calls sit inside async functions but never awaited. None compiled. The package's CI never builds IntegrationTesting. The only job that builds it, `integration_build_xcode27`, is disabled with `if: false`, so ml-explore#579 merged despite breaking this project.
…l-explore#602) loraLayers returned only selfAttn, so adapters targeting mlp.* failed LoRAContainer.load with unhandled keys. Gemma3Text/Qwen35/Llama (and the Gemma4 VLM wrapper) return the decoder layers; match them. Adds a regression test that loads a synthetic self_attn + mlp adapter onto a tiny Gemma4TextModel. Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
github-actions
Bot
force-pushed
the
sync/upstream-latest
branch
4 times, most recently
from
September 9, 2026 09:05
6db4487 to
ca43b53
Compare
Support differentiation and activation checkpointing in gated-delta recurrence
• Enable training via ops fallback: The fused Metal CustomKernel lacks a VJP, causing fatal aborts during backward passes (e.g., in Qwen3.5/Qwen3-Next). Added a useKernel flag to gatedDeltaUpdate (defaults to true), passing !training from calling models to fall back to the differentiable ops path, matching Python mlx-lm. Added gradient sanity and numerical parity tests.
• Chunked activation recomputation: The step-by-step ops graph retains all intermediate states, consuming unviable amounts of memory at scale (~4 MB/step for Qwen3.5-27B). Wrapped recurrence in 16-step chunks via CustomFunction to recompute states during backward passes, mimicking mx.checkpoint.
• Inference unchanged: Continues using the fused kernel by default.
github-actions
Bot
force-pushed
the
sync/upstream-latest
branch
from
September 10, 2026 09:04
ca43b53 to
240b7ea
Compare
…ml-explore#613) NaiveStreamingDetokenizer.next() measured the common prefix between the previous decode and the new one in Characters, so a token that appended a combining scalar (U+FE0F, a zero-width joiner, an accent) to the previous character re-emitted the whole merged cluster: a ZWJ flag sequence streamed as its base flag repeated, a quote followed by a variation selector as two quotes. The prefix is now measured in Unicode scalars, which compares exactly and keeps the fallback for tokenizers whose decode is not append-only. Four tests in StreamingDetokenizerTests stream a ZWJ flag sequence, a variation selector after an ASCII quote, a combining accent, and a variation selector after a multi-byte character split across two tokens, and compare the scalar sequence with the full decode.
* Add configuration-based LoRA metadata discovery * Share LoRA metadata discovery across LLM and VLM factories
…l-explore#584) RotatingKVCache: make trim wrap-aware instead of corrupting the ring Speculative decoding rewinds rejected drafts by trimming, but trimming a wrapped ring previously left dead rows in the logical timeline. Trimming now linearizes the ring to temporal order and discards the newest rows, clamped to the non-pinned span. • Direct callers (speculative iterators): Accept documented, bounded window-edge loss rather than silent corruption when overwrites have occurred. • Exact callers (isTrimmable): Retain strict offset + positions < maxSize semantics for staged rounds, restore points, and prompt cache reuse. • State & layout: Decouple offset == fill by tracking live rows via idx and an explicit wrapped flag (serialized as a 7th metaState value; backward-compatible with 5/6-value legacy states). • Includes a fix for rotating cache trim following legacy restore.
github-actions
Bot
force-pushed
the
sync/upstream-latest
branch
from
September 11, 2026 09:02
240b7ea to
557f0ca
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated PR to synchronize SharpAI fork with Apple mlx-swift-lm upstream repository. Please review merge conflicts (if any) and verify CI pipelines.
Opened manually — the workflow's own
gh pr createstep has been failing silently every day since this branch started diverging (Resource not accessible by personal access token), so this backlog (229 commits) was never surfaced for review.SWIFTLM_PR_TOKENneeds its permissions fixed before the daily workflow can do this on its own again.🤖 Generated with Claude Code