Repository navigation

Add validated 60k DeepSeek V4 Flash LoRA training and serving - #84

Merged

Peyton Walters (pawalt) merged 112 commits into

mainfrom

devin/1780686358-deepseek-v4-flash-sft

Jul 19, 2026

Merged

Add validated 60k DeepSeek V4 Flash LoRA training and serving#84
Peyton Walters (pawalt) merged 112 commits into
mainfrom
devin/1780686358-deepseek-v4-flash-sft

Commits on Jun 5, 2026
- Add DeepSeek V4 Flash SFT example
  devin-ai-integration[bot]
  committed
  897e727
- Pin DeepSeek SFT image dependency
  devin-ai-integration[bot]
  committed
  024450e
- Add DeepSeek SFT smoke training controls
  devin-ai-integration[bot]
  committed
  7c2f698
- Make DeepSeek SFT wandb secret optional
  devin-ai-integration[bot]
  committed
  2157fa5
- Use cluster-stable DeepSeek SFT run IDs
  devin-ai-integration[bot]
  committed
  13bf269
- Resume DeepSeek SFT from checkpoints
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  739e88d
- Add mcore-bridge dep and fix README smoke data-folder
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  f0b1f9f
- Fix DeepSeek SFT checkpoint reload and CLI args
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  635d2a9
- Upgrade transformers to 5.10.2 for native deepseek_v4 support; fix README ordering
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  17c5732
- Install mcore-bridge with --no-deps to resolve transformers 5.10.2 conflict
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  1246b9c
- Patch transformers 4.57.4 to recognize DeepSeek V4 config
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  b17a6ed
- Verify DeepSeek V4 config registration at image build time
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  0419e2c
- Split DeepSeek V4 config verification into separate image step
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  245dab9
- Patch transformers AutoConfig module for DeepSeek V4
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  724bba1
- Populate DeepSeek V4 config defaults for mcore bridge
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  894e062
- Skip default instantiation for DeepSeek V4 config shim
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  6a80d85
- Make default DeepSeek run IDs retry-stable
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  9c1e43e
- Set DeepSeek MoE router score for Megatron
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  55f668b
- Pass DeepSeek router score through Megatron kwargs
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  4078b52
- Preserve base Megatron while adding mcore bridge
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  7833fef
- Use Megatron Core dev for DSv4 hybrid attention
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  387865e
- Avoid core attention recompute for DSv4 hybrid
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  2bd3518
- Patch DSv4 rope fields for Megatron Core dev
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  5ffb4d4
- Set DSv4 YaRN original context default
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  f9b68fc
- Make W&B training entrypoint optional
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  fda1139
- Disable padding-free batches for DSv4 CSA
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  92c8590
- Save DSv4 smoke checkpoint in Megatron format
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  c1ae86e
- Format DeepSeek SFT entrypoint
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  a65d107
Commits on Jun 8, 2026
- Add export_and_eval: checkpoint export → vLLM deploy → GSM8K eval
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  ad411c9
- Fix: rewrite export_and_eval without heredoc corruption
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  c11d5aa
- Fix export/eval checkpoint detection and server logging
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  8b54430
- Ensure export eval cleans up vLLM server on errors
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  376e740
- Verify DeepSeek mcore bridge patch during image build
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  faddb1f
- Register DeepSeek V4 CausalLM mapping for export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  b31ffce
- Fix DeepSeek V4 auto model patch registration
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  95c2dee
- Allow DeepSeek V4 config default construction
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  fd4388a
- Use vLLM transformers fallback for DeepSeek V4 export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  bfb33a9
- Improve DeepSeek V4 vLLM startup diagnostics
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  651b629
- Compare GSM8K answers numerically
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  dd0feac
- Retry DeepSeek V4 export eval on transient failures
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  d8e361e
- Serve DeepSeek V4 export with latest vLLM
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  0ca1474
- Use python3 in vLLM image setup
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  df2b481
- Declare Python for vLLM serving image
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  d560e8e
- Clear vLLM image entrypoint for Modal
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  a1d2e0e
- Preserve vLLM image Python environment
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  1f7aa79
- Update vLLM request logging flag
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  ae9e0d6
- Sanitize DeepSeek V4 config for vLLM
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  847512e
- Add DeepSeek V4 compression ratios for vLLM
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  ebf5b44
- Use Triton MoE backend for BF16 export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  6f12c0d
- Restore DeepSeek V4 quantization config
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  1e8c9f2
- Use fp8 KV cache for DeepSeek V4 serving
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  1af434b
- Backfill DeepSeek V4 Flash config fields
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  0c2f9af
- Use MegaMoE expert-parallel serving
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  e1b4a16
- Serve BF16 export with Transformers backend
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  2768d08
- Set pad token for DeepSeek V4 serving
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  9ed8209
- Add DeepSeek V4 config aliases for serving
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  62a716a
- Add missing DeepSeek V4 serving defaults
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  0578668
- Use native vLLM for BF16 DeepSeek export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  d36a1de
- Patch vLLM DeepSeek V4 BF16 serving
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  c062af8
- Thread model parallel sizes through export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  d7d5d97
- Use BF16 attention path for merged DeepSeek export
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  efd6215
- Fix DeepSeek vLLM deploy eval path
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  a14fc3c
- Align DeepSeek deploy image with validation run
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  8033ccb
- Wait for vLLM server cleanup
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  26392b0
- Handle torch_dtype in vLLM config patch
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  e1a61c8
Commits on Jun 9, 2026
- Add long-context (60k) meeting summarization loop
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  6573ffe
- Avoid mutating cached base model config during eval
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  eed8e9b
- Use enough long-context examples for DP smoke runs
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  2859713
- Enable sequence parallel for CP training
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  381303b
- Fix long-context export and vLLM serving path
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  a2c4428
- Disable sequence parallel for CP-only training
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  049e635
- Fix multi-node export post-processing
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  fa6336d
- Add pipeline layout for PP training
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  5c9a88d
- Patch RoPE length handling for CP smoke
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  d2ac38e
- Make RoPE patch match pinned Megatron
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  1c03e85
- Make RoPE CP patch robust
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  b84cf6e
- Serialize multi-node export volume commit
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  6a78e06
Commits on Jun 10, 2026
- Validate DeepSeek long-context SFT loop
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  045ab44
- Preserve gradients in long-context DeepSeek patches
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  0706497
- Reduce autograd-safe long-context memory
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  e09ebbb
- Stabilize DeepSeek long-context LoRA target
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  b1c5253
- Extract DeepSeek patches into documented registry
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  e77cd01
- Split combined mcore_bridge patch and add ablation rationale
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  19a1462
- Disable W&B reporting explicitly by default
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  434e942
- Document DeepSeek patch ablation evidence
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  26a7526
Commits on Jun 11, 2026
- Add plan for 60k context testing without gradient-unsafe patches
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  3ebaeb2
- Guard DeepSeek SFT target modules
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  8af9a0c
- Expand DeepSeek target module guard
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  fbaae62
- DSv4-Flash 60k: drop the memory patches, commit to CP=4 default (#87)
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  authored
  eb03fb6
- Clean up DeepSeek long-context docs
  devin-ai-integration[bot]
  committed
  a9cc673
Commits on Jun 12, 2026
Commits on Jun 15, 2026
- Document 60k patches-removed path as unvalidated (OOMs)
  
  devin-ai-integration[bot]
  and
  Peyton Walters (pawalt)
  committed
  c199377
Commits on Jul 17, 2026
Commits on Jul 18, 2026
- Validate 60k LoRA training on 8 H200 nodes
  Peyton Walters (pawalt)
  committed
  e363adb
Commits on Jul 19, 2026
- Clean up DeepSeek V4 training example
  Peyton Walters (pawalt)
  committed
  777ade9

Commits

Commits on Jun 5, 2026

Commits on Jun 8, 2026

Commits on Jun 9, 2026

Commits on Jun 10, 2026

Commits on Jun 11, 2026

Commits on Jun 12, 2026

Commits on Jun 15, 2026

Commits on Jul 17, 2026

Commits on Jul 18, 2026

Commits on Jul 19, 2026