fix(mps): 3-layer defense against Metal command buffer crash (1.3B + 5B)#1776
Merged
deepbeepmeep merged 1 commit intoMay 10, 2026
Merged
Conversation
…and 5B Root cause: MPS fallback ops corrupt Metal command buffers when mixed with native MPS SDPA attention. Affects Wan 2.1 1.3B and 2.2 5B. 3-layer defense: 1. torch.mps.synchronize() before every SDPA call 2. .contiguous() after .to() and .expand() in attention.py 3. try/except with manual matmul+softmax fallback Plus periodic empty_cache every 256 SDPA calls.
This was referenced May 8, 2026
|
@huangyebiaoke Is there a particular Profile that would be best to test with? |
Author
|
@SquishedSquirrel Good question — for the 3-layer defense testing, stick with Profile 5 for now. It's slower but safest (lowest memory pressure), so if it crashes we know it's definitely the command buffer issue, not OOM. Once Profile 5 is confirmed stable with the fix, we can try Profile 4 and 3+ to push memory limits. Also: please use Euler sampler — UniPC and DPM++ produce corrupted output on MPS regardless of the fix. And apply both #1777 (already merged) + #1776 together — I left detailed steps on the main thread #1750. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
@SquishedSquirrel found that Wan 2.1 1.3B crashes on M3 Max 36GB with:
This happens on all memory profiles (3, 3+, 4, 5) — while 14B Lightning works fine.
Root Cause
The existing fix (forcing SDPA MATH backend) was not enough. MPS fallback ops (CPU fallback from
PYTORCH_ENABLE_MPS_FALLBACK=1) corrupt Metal command buffers when mixed with native MPS SDPA attention. This affects:Fix: 3-Layer Defense
Plus periodic empty_cache every 256 SDPA calls to prevent memory fragmentation.
Testing Needed
@SquishedSquirrel @cn0ss — please re-test on your Macs:
Supersedes: #1773 (absorbs its fixes + adds fallback)