[wirybeaver]

Summary

BF16 MoonEP PoC integration for the Kimi-K3 roadmap:

• recognizes --moe-a2a-backend moonep as a distinct backend;
• adds MoonEP dispatch/combine data contracts and a process-wide buffer facade;
• adds BF16 global expert weight layout + correctness-first expert segment runner;
• wires eager dispatch -> prefetch -> compute -> combine runtime path;
• adds a distributed BF16 validation harness.

Validation

Local:

• git diff --check
• python3 -m compileall on touched Python files
• cargo check attempted but local cargo is not installed

Remote Vast.ai:

• 4x H100 SXM (NV6) rejected as insufficient: CUDA multicast unsupported.
• 8x H100 SXM (NV18) passed BF16 validation on 8 ranks:
  • global_ok=true
  • max_abs_err=0.0
  • relative_err=0.0
  • shape: tokens=128, hidden_size=1024, intermediate_size=128, top_k=2, num_experts=16, num_prefetch_slots=2

Docker/NCU:

• Ran docker-ncu-benchmark preflight on suitable H100 hosts.
• Vast.ai H100 NVSwitch/multicast offers were container instances without Docker/NVIDIA Container Toolkit, so Docker GPU injection could not be tested there.
• Host NCU was present but counters were restricted (ERR_NVGPUCTRPERM), so no NCU performance claims are made.

Artifacts are recorded locally under .scratch/moonep/remote-validation-20260802T050429Z/.

Limitations

• BF16/unquantized only.
• Eager-only; CUDA graph disabled for MoonEP PoC.
• Correctness-first Python segment loop, not final grouped GEMM performance kernel.
• PoC global expert-row storage is memory-heavy; production Kimi-K3 needs true sharded symmetric-memory expert row ownership.
• Requires NVSwitch/multicast-capable hosts (8x H100/H200/B200 class); multi-RTX 5090 without NVLink is not useful for MoonEP all-to-all correctness.

Closes #16.

---

CI States

Latest PR Test (Base): ❌ Run #30734488332
Latest PR Test (Extra): ❌ Run #30734488294

[wirybeaver]

Superseded by upstream draft PR sgl-project#33249.