Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions performance/checkpointing.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -135,9 +135,9 @@ The most common checkpoint failure is vLLM's default `fork` start method. Set `V

The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/).

<!-- The headings below are linked to by anchor from the checkpoint agent's
{/_ The headings below are linked to by anchor from the checkpoint agent's
error messages (internal/agent/checkpoint_error.go). Renaming one breaks the
link customers are told to follow. -->
link customers are told to follow. _/}

## When a checkpoint fails

Expand All @@ -152,9 +152,9 @@ The log line matters because most apps fire the request and never read the respo

## vLLM uses fork by default

vLLM on Linux starts worker processes with `fork`. That is the default — you did not have to change anything to hit this. Forked workers inherit the parent's CUDA context, so CUDA is still live in a child process when we snapshot, and the checkpoint fails.
vLLM on Linux starts worker processes with `fork`. That is the default. You did not have to change anything to hit this. Forked workers inherit the parent's CUDA context, so CUDA is still live in a child process when we snapshot, and the checkpoint fails.

Set the start method to `spawn` **before** vLLM is imported:
Set the start method to `spawn` **before** importing vLLM:

```python
import os
Expand All @@ -166,7 +166,7 @@ from vllm import AsyncLLMEngine

If vLLM is imported at module load, before your own code runs, set the same value as an [app secret](/other-topics/using-secrets) named `VLLM_WORKER_MULTIPROC_METHOD` with value `spawn`, then redeploy.

This is the most common checkpoint failure. Other GPU servers that use Python `multiprocessing` with `fork` have the same incompatibility — switch them to `spawn` as well.
This is the most common checkpoint failure. Other GPU servers that use Python `multiprocessing` with `fork` have the same incompatibility. Switch them to `spawn` as well.

Redeploy after the change, wait for the new container to finish warming up, and call the checkpoint endpoint again.

Expand All @@ -182,10 +182,10 @@ A checkpoint cannot be taken while a process outside the snapshot still holds an

CUDA Unified Memory (UVM, `cudaMallocManaged`) cannot be checkpointed. Unified allocations are managed by the driver across host and device, and the checkpointer has no way to capture that state, so it refuses the snapshot.

Use ordinary device allocations instead `cudaMalloc`, or PyTorch tensors moved to CUDA with `.to("cuda")`. If a dependency enables unified memory on your behalf, disable that option; if it cannot be disabled, the app cannot be checkpointed today.
Use ordinary device allocations instead: `cudaMalloc`, or PyTorch tensors moved to CUDA with `.to("cuda")`. If a dependency enables unified memory on your behalf, disable that option; if it cannot be disabled, the app cannot be checkpointed today.

## Unknown cause

The failure is not one of the causes above. The message says `checkpoint failed: unknown cause` rather than inventing a reason or returning a bare error.

Some apps are not compatible yet, and some failure modes still need classifying. Email [support@cerebrium.ai](mailto:support@cerebrium.ai) with your app id and the time of the failed checkpoint so we can match it against the node logs and, where we can, turn it into a named cause on this page.
Some apps are not compatible yet, and some failure modes still need classifying. Email [support@cerebrium.ai](mailto:support@cerebrium.ai) with your app ID and the time of the failed checkpoint. That lets us match it against the node logs and, where we can, turn it into a named cause on this page.
10 changes: 5 additions & 5 deletions v4/examples/langchain-langsmith.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def exponentiate(x: float, y: float) -> float:
return x**y
```

`AIMessage.tool_calls`: An attribute on AIMessage that provides easy access to model-initiated tool calls, specifying invocations in the bind_tools format:
`AIMessage.tool_calls`: An attribute on AIMessage that provides easy access to model-initiated tool calls, specifying invocations in the `bind_tools` format:

```python
# -> AIMessage(
Expand Down Expand Up @@ -214,7 +214,7 @@ The code above:
3. Uses `find_available_slots` helper function to format Cal.com API responses into readable time slots

The book_slot tool follows a similar pattern. It books a slot based on the selected time/day. Get the eventTypeId from the dashboard by selecting an event and grabbing the ID from the URL.
The `book_slot` tool follows a similar pattern. It books a slot based on the selected time/day. Get the `eventTypeId` from the dashboard by selecting an event and grabbing the ID from the URL.

```python
@tool
Expand Down Expand Up @@ -286,9 +286,9 @@ The agent executor consists of:

### Chatbot Setup

The above code only handles a single question. Finding a mutually suitable time requires a multi-turn conversation. LangChain’s RunnableWithMessageHistory() adds tool calling capabilities and message memory.
The above code only handles a single question. Finding a mutually suitable time requires a multi-turn conversation. LangChain’s `RunnableWithMessageHistory()` adds tool calling capabilities and message memory.

It stores previous replies in the chat_history variable (from the prompt template) and ties them to a session identifier, so the API remembers information per user/session:
It stores previous replies in the `chat_history` variable (from the prompt template) and ties them to a session identifier, so the API remembers information per user/session:

```python
from langchain.memory import ChatMessageHistory
Expand Down Expand Up @@ -335,7 +335,7 @@ if __name__ == "__main__":
This code:

- Defines a Pydantic object specifying the expected API parameters: user prompt and session ID.
- The predict function (Cerebrium’s API entry point) passes the prompt and session ID to the agent and returns results.
- The `predict` function (Cerebrium’s API entry point) passes the prompt and session ID to the agent and returns results.

Install pip dependencies locally: `pip install pydantic langchain pytz openai langchain_openai langchain-community`, then run `python main.py`. Replace secrets with actual values when running locally. Output looks similar to:
Expand Down