Skip to content

docs: explain why a checkpoint failed and how to fix it - #335

Merged
Hkhan161 merged 9 commits into
masterfrom
harris/checkpointing-failure-docs
Aug 31, 2026
Merged

docs: explain why a checkpoint failed and how to fix it#335
Hkhan161 merged 9 commits into
masterfrom
harris/checkpointing-failure-docs

Conversation

@Hkhan161

@Hkhan161 Hkhan161 commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

Customers hitting a checkpoint failure had nothing to read. Master already shipped a Performance guide for checkpointing (performance/checkpointing); this PR adds the failure-cause sections onto that page rather than introducing a duplicate.

Covers the three causes the checkpoint agent can name, with the fix for each:

  • vLLM uses fork by default — set VLLM_WORKER_MULTIPROC_METHOD=spawn before vLLM is imported, or as an app secret when it is imported at module load.
  • Open GPU file descriptors — prefer spawn for any process touching the GPU, don't init CUDA then fork.
  • CUDA Unified MemorycudaMallocManaged can't be checkpointed; use ordinary device allocations.
  • Unknown cause — what to send support so we can classify it.

Also documents where the failure shows up (response body and the CEREBRIUM_CHECKPOINT_FAILED log line), and adds spawn to the existing vLLM example so the sample itself does not hit the most common failure.

The agent's error messages deep-link to these headings by anchor (#vllm-uses-fork-by-default, #open-gpu-file-descriptors, #cuda-unified-memory, #unknown-cause). Paired with CerebriumAI/cerebrium-checkpoint-agent#95 — merge this first so the links resolve.

Test plan

  • mintlify dev renders /performance/checkpointing and the new sections appear
  • Anchors resolve: #vllm-uses-fork-by-default, #open-gpu-file-descriptors, #cuda-unified-memory, #unknown-cause
  • Relative link to /other-topics/using-secrets resolves

Hkhan161 and others added 2 commits February 26, 2026 12:06
Customers hitting a checkpoint failure had nothing to read: the agent
returned a bare error and no page described the known incompatibilities.

Documents the three causes the agent can name (vLLM's default fork start
method, open GPU file descriptors, CUDA Unified Memory) with the fix for
each, plus where to find the failure. The headings are anchor targets from
the agent's error messages, so they carry a comment against renaming.

Co-authored-by: Cursor <cursoragent@cursor.com>
@mintlify

mintlify Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
cerebrium 🟢 Ready View Preview Aug 31, 2026, 3:35 PM

Hkhan161 and others added 5 commits August 31, 2026 11:40
…eckpointing

Master renamed the site (dropped the cerebrium/ prefix) and already shipped a
Performance guide for checkpointing. Keep the failure-cause sections there so
the agent's heading anchors still resolve, and drop the duplicate other-topics
page. Also take master's private-registry copy and Other concepts nav.

Co-authored-by: Cursor <cursoragent@cursor.com>
The bot formatted the old other-topics pages, which this branch already
dropped in favour of performance/checkpointing.

Co-authored-by: Cursor <cursoragent@cursor.com>
That content lived on the local branch we started from and is not on
master. This PR should only add the checkpoint failure sections.

Co-authored-by: Cursor <cursoragent@cursor.com>
JSX comments in this MDX file get rewritten as {/_ ... _/}, which would
render on the page. Use an HTML comment instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Hkhan161
Hkhan161 merged commit be20a1e into master Aug 31, 2026
1 of 2 checks passed
@Hkhan161
Hkhan161 deleted the harris/checkpointing-failure-docs branch August 31, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants