diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index 4eeb2b8b..34ccf7e3 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -135,9 +135,9 @@ The most common checkpoint failure is vLLM's default `fork` start method. Set `V The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). - +link customers are told to follow. _/} ## When a checkpoint fails @@ -152,9 +152,9 @@ The log line matters because most apps fire the request and never read the respo ## vLLM uses fork by default -vLLM on Linux starts worker processes with `fork`. That is the default — you did not have to change anything to hit this. Forked workers inherit the parent's CUDA context, so CUDA is still live in a child process when we snapshot, and the checkpoint fails. +vLLM on Linux starts worker processes with `fork`. That is the default. You did not have to change anything to hit this. Forked workers inherit the parent's CUDA context, so CUDA is still live in a child process when we snapshot, and the checkpoint fails. -Set the start method to `spawn` **before** vLLM is imported: +Set the start method to `spawn` **before** importing vLLM: ```python import os @@ -166,7 +166,7 @@ from vllm import AsyncLLMEngine If vLLM is imported at module load, before your own code runs, set the same value as an [app secret](/other-topics/using-secrets) named `VLLM_WORKER_MULTIPROC_METHOD` with value `spawn`, then redeploy. -This is the most common checkpoint failure. Other GPU servers that use Python `multiprocessing` with `fork` have the same incompatibility — switch them to `spawn` as well. +This is the most common checkpoint failure. Other GPU servers that use Python `multiprocessing` with `fork` have the same incompatibility. Switch them to `spawn` as well. Redeploy after the change, wait for the new container to finish warming up, and call the checkpoint endpoint again. @@ -182,10 +182,10 @@ A checkpoint cannot be taken while a process outside the snapshot still holds an CUDA Unified Memory (UVM, `cudaMallocManaged`) cannot be checkpointed. Unified allocations are managed by the driver across host and device, and the checkpointer has no way to capture that state, so it refuses the snapshot. -Use ordinary device allocations instead — `cudaMalloc`, or PyTorch tensors moved to CUDA with `.to("cuda")`. If a dependency enables unified memory on your behalf, disable that option; if it cannot be disabled, the app cannot be checkpointed today. +Use ordinary device allocations instead: `cudaMalloc`, or PyTorch tensors moved to CUDA with `.to("cuda")`. If a dependency enables unified memory on your behalf, disable that option; if it cannot be disabled, the app cannot be checkpointed today. ## Unknown cause The failure is not one of the causes above. The message says `checkpoint failed: unknown cause` rather than inventing a reason or returning a bare error. -Some apps are not compatible yet, and some failure modes still need classifying. Email [support@cerebrium.ai](mailto:support@cerebrium.ai) with your app id and the time of the failed checkpoint so we can match it against the node logs and, where we can, turn it into a named cause on this page. +Some apps are not compatible yet, and some failure modes still need classifying. Email [support@cerebrium.ai](mailto:support@cerebrium.ai) with your app ID and the time of the failed checkpoint. That lets us match it against the node logs and, where we can, turn it into a named cause on this page. diff --git a/v4/examples/langchain-langsmith.mdx b/v4/examples/langchain-langsmith.mdx index 696922a8..25837aab 100644 --- a/v4/examples/langchain-langsmith.mdx +++ b/v4/examples/langchain-langsmith.mdx @@ -24,7 +24,7 @@ def exponentiate(x: float, y: float) -> float: return x**y ``` -`AIMessage.tool_calls`: An attribute on AIMessage that provides easy access to model-initiated tool calls, specifying invocations in the bind_tools format: +`AIMessage.tool_calls`: An attribute on AIMessage that provides easy access to model-initiated tool calls, specifying invocations in the `bind_tools` format: ```python # -> AIMessage( @@ -214,7 +214,7 @@ The code above: 3. Uses `find_available_slots` helper function to format Cal.com API responses into readable time slots ‍ -The book_slot tool follows a similar pattern. It books a slot based on the selected time/day. Get the eventTypeId from the dashboard by selecting an event and grabbing the ID from the URL. +The `book_slot` tool follows a similar pattern. It books a slot based on the selected time/day. Get the `eventTypeId` from the dashboard by selecting an event and grabbing the ID from the URL. ```python @tool @@ -286,9 +286,9 @@ The agent executor consists of: ### Chatbot Setup -The above code only handles a single question. Finding a mutually suitable time requires a multi-turn conversation. LangChain’s RunnableWithMessageHistory() adds tool calling capabilities and message memory. +The above code only handles a single question. Finding a mutually suitable time requires a multi-turn conversation. LangChain’s `RunnableWithMessageHistory()` adds tool calling capabilities and message memory. -It stores previous replies in the chat_history variable (from the prompt template) and ties them to a session identifier, so the API remembers information per user/session: +It stores previous replies in the `chat_history` variable (from the prompt template) and ties them to a session identifier, so the API remembers information per user/session: ```python from langchain.memory import ChatMessageHistory @@ -335,7 +335,7 @@ if __name__ == "__main__": This code: - Defines a Pydantic object specifying the expected API parameters: user prompt and session ID. -- The predict function (Cerebrium’s API entry point) passes the prompt and session ID to the agent and returns results. +- The `predict` function (Cerebrium’s API entry point) passes the prompt and session ID to the agent and returns results. ‍ Install pip dependencies locally: `pip install pydantic langchain pytz openai langchain_openai langchain-community`, then run `python main.py`. Replace secrets with actual values when running locally. Output looks similar to: