Context
In MoE models (e.g., DeepSeek-R1), expert parameters are 97.45% of total but only 3.13% activated per token. OD-MoE (Dec 2025) achieves 99.94% expert prediction accuracy with 1/3 GPU memory.
Proposal
- Store MoE expert weights in OPFS (3-4x faster I/O than IndexedDB)
- Predict which experts are needed 1-2 layers ahead
- Load only active experts into GPU memory on-demand
- Run much larger models (8x22B Mixtral) by only keeping ~2 experts resident
- MoLE (ICML 2025): pre-computed lookup tables for experts — zero computation for expert layers
Impact
- Run 10B+ effective-parameter MoE models in browser with <2GB active memory
- First browser engine with native MoE support
References
- OD-MoE — on-demand expert loading
- FlashMoE — ML-based cache replacement
- MoLE — lookup table experts
Context
In MoE models (e.g., DeepSeek-R1), expert parameters are 97.45% of total but only 3.13% activated per token. OD-MoE (Dec 2025) achieves 99.94% expert prediction accuracy with 1/3 GPU memory.
Proposal
Impact
References