Skip to content

fix(mps): 3-layer defense against Metal command buffer crash (1.3B + 5B)#1776

Merged
deepbeepmeep merged 1 commit into
deepbeepmeep:MPSfrom
huangyebiaoke:fix/mps-1-3b-command-buffer
May 10, 2026
Merged

fix(mps): 3-layer defense against Metal command buffer crash (1.3B + 5B)#1776
deepbeepmeep merged 1 commit into
deepbeepmeep:MPSfrom
huangyebiaoke:fix/mps-1-3b-command-buffer

Conversation

@huangyebiaoke

@huangyebiaoke huangyebiaoke commented May 8, 2026

Copy link
Copy Markdown

Problem

@SquishedSquirrel found that Wan 2.1 1.3B crashes on M3 Max 36GB with:

[IOGPUMetalCommandBuffer validate]:213: failed assertion commit an already committed command buffer

This happens on all memory profiles (3, 3+, 4, 5) — while 14B Lightning works fine.

Root Cause

The existing fix (forcing SDPA MATH backend) was not enough. MPS fallback ops (CPU fallback from PYTORCH_ENABLE_MPS_FALLBACK=1) corrupt Metal command buffers when mixed with native MPS SDPA attention. This affects:

  • Wan 2.1 1.3B (standard safetensors — fallback ops from layer norm, activations)
  • Wan 2.2 5B (quanto mbf16 quantization)

Fix: 3-Layer Defense

  1. torch.mps.synchronize() before every SDPA call — flushes pending fallback ops
  2. .contiguous() after .to() and .expand() in attention.py — tensor buffer isolation
  3. try/except + manual fallback (matmul+softmax) if SDPA fails — catches macOS 26.x Metal bugs

Plus periodic empty_cache every 256 SDPA calls to prevent memory fragmentation.

Testing Needed

@SquishedSquirrel @cn0ss — please re-test on your Macs:

  • Wan 2.1 1.3B on M3/M4 Max
  • Wan 2.2 5B (with and without fastwan)
  • Verify 14B still works

Supersedes: #1773 (absorbs its fixes + adds fallback)

…and 5B

Root cause: MPS fallback ops corrupt Metal command buffers when mixed
with native MPS SDPA attention. Affects Wan 2.1 1.3B and 2.2 5B.

3-layer defense:
1. torch.mps.synchronize() before every SDPA call
2. .contiguous() after .to() and .expand() in attention.py
3. try/except with manual matmul+softmax fallback

Plus periodic empty_cache every 256 SDPA calls.
@SquishedSquirrel

Copy link
Copy Markdown

@huangyebiaoke Is there a particular Profile that would be best to test with?

@huangyebiaoke

Copy link
Copy Markdown
Author

@SquishedSquirrel Good question — for the 3-layer defense testing, stick with Profile 5 for now. It's slower but safest (lowest memory pressure), so if it crashes we know it's definitely the command buffer issue, not OOM.

Once Profile 5 is confirmed stable with the fix, we can try Profile 4 and 3+ to push memory limits.

Also: please use Euler sampler — UniPC and DPM++ produce corrupted output on MPS regardless of the fix. And apply both #1777 (already merged) + #1776 together — I left detailed steps on the main thread #1750.

@deepbeepmeep deepbeepmeep merged commit 12fcfed into deepbeepmeep:MPS May 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants