Skip to content

Stabilize WAN MPS generation on Apple Silicon#1868

Open
SquishedSquirrel wants to merge 1 commit into
deepbeepmeep:mainfrom
SquishedSquirrel:mps-wan-stability-current
Open

Stabilize WAN MPS generation on Apple Silicon#1868
SquishedSquirrel wants to merge 1 commit into
deepbeepmeep:mainfrom
SquishedSquirrel:mps-wan-stability-current

Conversation

@SquishedSquirrel

@SquishedSquirrel SquishedSquirrel commented Jun 5, 2026

Copy link
Copy Markdown

This PR improves WAN generation stability on Apple Silicon/MPS, especially for WAN 2.2 5B and WAN 2.1 1.3B.

Changes:

  • Default MPS SDPA to a synchronized manual fallback, with native SDPA opt-in via WAN2GP_MPS_NATIVE_SDPA=1.
  • Add WAN-specific MPS synchronization/cache cleanup boundaries around transformer calls, sampler steps, generation boundaries, and cache clearing.
  • Run quantized WAN WebUI generation inline on MPS to avoid worker-thread Metal command-buffer crashes.
  • Disable live WAN latent preview frames by default on MPS while preserving progress updates.
  • Keep diagnostic env switches for native SDPA, live previews, inline worker behavior, and joint CFG.

Validation:

  • WAN 2.2 5B passed multiple WebUI generations.
  • WAN 2.1 1.3B passed multiple WebUI generations.
  • Selected 14B smoke test passed.
  • Encode/post-processing sanity check passed.

Let me be clear that these patches were done with Codex and GPT 5.5. I can't speak to the quality or durability of the patches. They center around WAN, as that was my primary goal. Memory usage is quite high, so a 36GB Mac or better is really needed if you plan to do anything with 14B, and even then the frame size has to be pretty small.

Manual fallback SDPA was chosen for lower memory impact and slightly faster speed.

WebUI worker-thread note

During testing, the quantized WAN transformer path behaved differently in headless queue processing versus the Gradio/WebUI path.

The true quantized WAN 2.2 5B checkpoint could complete repeated headless generations, but the same model/configuration would intermittently crash from the WebUI with Metal command-buffer assertions such as:

  • commit an already committed command buffer
  • commit command buffer with uncommitted encoder

The instability appeared when generation was dispatched through the extra WebUI async_run_in("generation", ...) worker thread. Running the queued WAN generation inline in the Gradio request path for quantized WAN transformer tasks avoided those crashes in local testing.

This is intentionally scoped narrowly:

  • only on MPS
  • only WAN video tasks
  • only when the selected transformer checkpoint appears quantized (quanto, int8, or fp8)
  • overrideable with WAN2GP_MPS_WEBUI_INLINE_WORKER

This keeps the WebUI behavior unchanged for CUDA and for non-quantized/non-WAN tasks.

@deepbeepmeep

Copy link
Copy Markdown
Owner

Thx for this work but in order to integrate in WanGP I need to minimize the potential impact on the existence not only in terms of compatibility but also for future maintenance:

  • we should avoid doing change that are unlikely to improve mps compatibility and complexify code at the same time, for instance: joint pass and gradio async queue.
  • I dont understand all the sdpa fallback in any2video since sdpa is the default attention with mps
  • changes to any2video are quite intrusive, any chance to get the same results with monkeypatches?

@SquishedSquirrel

Copy link
Copy Markdown
Author

There are two forms of sdpa, native and manual attention. Native sdpa was causing higher memory usage, and giving slower generations as well as errors. I will see what can be done.

@SquishedSquirrel

SquishedSquirrel commented Jun 18, 2026

Copy link
Copy Markdown
Author

@deepbeepmeep I have to bow out here. I understand what it is doing, but not at a code level, so I really don't think it wise that I keep trying to make patches that just "work" rather than patches that take into account the whole picture instead of just WAN.

For anyone who wants to pick up the torch (pun intended) the primary issue was that the code to support the gradio UI runs in another thread, and the monkey patches combined with MPS caused a race condition between the two threads which generates numerous command buffer and uncommitted encoder errors. This wasn't found until very late in trying to solve the issue, so even though Codex thinks it is a bad idea to start over and target that issue first, my gut says that might be the better solution.... solve the thread race condition first, then clean up what is left. Test by running queues in headless mode (create the queue in the UI, exit, and then run them headless and then compare completion with running the same task within the ui.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants