docs: explain why a checkpoint failed and how to fix it - #335
Merged
Conversation
…tries Made-with: Cursor
Customers hitting a checkpoint failure had nothing to read: the agent returned a bare error and no page described the known incompatibilities. Documents the three causes the agent can name (vLLM's default fork start method, open GPU file descriptors, CUDA Unified Memory) with the fix for each, plus where to find the failure. The headings are anchor targets from the agent's error messages, so they carry a comment against renaming. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
|
…eckpointing Master renamed the site (dropped the cerebrium/ prefix) and already shipped a Performance guide for checkpointing. Keep the failure-cause sections there so the agent's heading anchors still resolve, and drop the duplicate other-topics page. Also take master's private-registry copy and Other concepts nav. Co-authored-by: Cursor <cursoragent@cursor.com>
The bot formatted the old other-topics pages, which this branch already dropped in favour of performance/checkpointing. Co-authored-by: Cursor <cursoragent@cursor.com>
That content lived on the local branch we started from and is not on master. This PR should only add the checkpoint failure sections. Co-authored-by: Cursor <cursoragent@cursor.com>
…s' into harris/checkpointing-failure-docs
jonoirwinrsa
approved these changes
Aug 31, 2026
JSX comments in this MDX file get rewritten as {/_ ... _/}, which would
render on the page. Use an HTML comment instead.
Co-authored-by: Cursor <cursoragent@cursor.com>
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Customers hitting a checkpoint failure had nothing to read. Master already shipped a Performance guide for checkpointing (
performance/checkpointing); this PR adds the failure-cause sections onto that page rather than introducing a duplicate.Covers the three causes the checkpoint agent can name, with the fix for each:
VLLM_WORKER_MULTIPROC_METHOD=spawnbefore vLLM is imported, or as an app secret when it is imported at module load.spawnfor any process touching the GPU, don't init CUDA then fork.cudaMallocManagedcan't be checkpointed; use ordinary device allocations.Also documents where the failure shows up (response body and the
CEREBRIUM_CHECKPOINT_FAILEDlog line), and addsspawnto the existing vLLM example so the sample itself does not hit the most common failure.The agent's error messages deep-link to these headings by anchor (
#vllm-uses-fork-by-default,#open-gpu-file-descriptors,#cuda-unified-memory,#unknown-cause). Paired with CerebriumAI/cerebrium-checkpoint-agent#95 — merge this first so the links resolve.Test plan
mintlify devrenders/performance/checkpointingand the new sections appear#vllm-uses-fork-by-default,#open-gpu-file-descriptors,#cuda-unified-memory,#unknown-cause/other-topics/using-secretsresolves