From ba0d282e72058871d08bd42cc0f7bb7485d1e91d Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Wed, 26 Aug 2026 09:21:20 +0000 Subject: [PATCH 1/3] fix: vendor references/config.md so skill.md relative link resolves --- .prettierignore | 4 + references/config.md | 202 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+) create mode 100644 references/config.md diff --git a/.prettierignore b/.prettierignore index 25325ea4..55192e68 100644 --- a/.prettierignore +++ b/.prettierignore @@ -4,3 +4,7 @@ available-hardware.mdx # A checker there fails that repo's CI on any drift, and Prettier re-wraps this file # if allowed to touch it. Edit the skill upstream, never here. /skill.md + +# Vendored from skills/cerebrium/references/config.md in CerebriumAI/cerebrium-skills so the +# relative references/config.md link inside skill.md resolves. Edit upstream, never here. +/references/config.md diff --git a/references/config.md b/references/config.md new file mode 100644 index 00000000..4ecf4776 --- /dev/null +++ b/references/config.md @@ -0,0 +1,202 @@ +# cerebrium.toml + +One file per app, read by `cerebrium deploy` and `cerebrium run`. + +Two rules to internalise before editing one: + +1. **A key you leave out is set to the default, not left alone.** A deploy replaces the whole app + record. If a previous deploy or `cerebrium apps scale` set `max_replicas = 20` and the TOML + you deploy has no `max_replicas`, the app goes back to 1. Keep every value that matters in the + file. +2. **A key the CLI does not recognise is not rejected locally.** A misspelled `max_replica` does + not fail the deploy in the CLI, it just does nothing there. The whole file is uploaded + verbatim, so the backend sees the key too and is free to reject it. Check spelling against the + tables below rather than trusting a successful deploy. A TOML that does not parse at all does + fail the deploy. + +Defaults below are what the API applies when the key is absent. Where a published table +disagrees, prefer these, and set anything that matters explicitly. + +## `[cerebrium.deployment]` + +| Key | Applied when omitted | Notes | +| --- | --- | --- | +| `name` | required | App name. | +| `python_version` | `3.11` | `3.10`, `3.11`, `3.12` and `3.13` are accepted. Changing it forces a full rebuild. | +| `disable_auth` | `true` | **`true` means the endpoint is public.** Set `false` for anything real. | +| `include` | `["./*", "main.py", "cerebrium.toml"]` | Keep model weights out of it. | +| `exclude` | `[".*"]` | | +| `shell_commands` | `[]` | Run at the end of the build. | +| `pre_build_commands` | `[]` | Run before dependencies install. | +| `docker_base_image_url` | `debian:bookworm-slim` | Changing it forces a full rebuild. | +| `use_uv` | `false` | uv instead of pip. Much faster on large dependency trees. | +| `deployment_initialization_timeout` | `600` | Seconds, 60 to 830. Raise it when weights load slowly. | + +`cerebrium init` scaffolds `disable_auth = true`. That is a convenience for a first curl, not a +production default. Treat flipping it to `false` as part of the first real deploy. + +## `[cerebrium.runtime]` + +| Key | Applied when omitted | Notes | +| --- | --- | --- | +| `container_runtime` | platform default | `"v1"` is runc, `"v2"` is gvisor. Any other value fails the deploy locally. | + +The sub-tables in the next two sections sit under the same heading and can be set alongside +`container_runtime`. + +## `[cerebrium.runtime.custom]` + +Only for a custom web server (FastAPI, ASGI, WebSockets, custom batching) or a Dockerfile build. +Omit the section to use the default Cortex runtime. + +| Key | Applied when omitted | Notes | +| --- | --- | --- | +| `port` | `8000` | Must match the port inside `entrypoint`. | +| `entrypoint` | required | A list (`["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]`) or a single string. The list form is unambiguous, prefer it. | +| `healthcheck_endpoint` | `""` (TCP ping) | Non-200 marks the instance unhealthy and restarts it. | +| `readycheck_endpoint` | `""` (TCP ping) | Non-200 removes the instance from routing. | +| `dockerfile_path` | unset | Build from a Dockerfile instead. The path must exist locally, and the image then owns its own dependencies and Python version. | + +## `[cerebrium.runtime.deepgram]` and `[cerebrium.runtime.rime]` + +Partner service runtimes. The table name is the partner, and an app uses one of them or neither. +Omit both unless you are deploying that partner's image. + +| Key | Applied when omitted | Notes | +| --- | --- | --- | +| `port` | partner default | Port the partner service listens on. | +| `model_name` | partner default | Partner model identifier, for example `arcana` or `mist`. | +| `language` | partner default | Language code, for example `en`. | + +```toml +[cerebrium.runtime.deepgram] +model_name = "arcana" +language = "en" +``` + +## `[cerebrium.hardware]` + +| Key | Applied when omitted | Accepted | +| --- | --- | --- | +| `cpu` | `2.0` | Depends on the compute type and the plan. See `references/hardware.md`. | +| `memory` | `4.0` | GB of host RAM, not VRAM. Same limits apply. | +| `compute` | `"CPU"` | One identifier, or a preference-ordered list of up to 5 from the same family. | +| `gpu_count` | `1` when `compute` is an accelerator, `0` for `CPU` | Per-type maximum, capped again by the plan. | +| `provider` | platform picks | `aws`, `crusoe` or `nebius`. | +| `region` | platform picks | See `references/hardware.md`. | + +## `[cerebrium.scaling]` + +| Key | Applied when omitted | Accepted | Notes | +| --- | --- | --- | --- | +| `min_replicas` | `0` | 0 to 2000, capped by plan | Above 0 keeps warm capacity and bills for it. | +| `max_replicas` | `1` | 1 to 2000, capped by plan | The most common cause of unexplained queueing. Raise it before load. | +| `replica_concurrency` | `1` on an accelerator, `100` on CPU | >= 1 | In-flight requests per replica. | +| `scaling_metric` | `concurrency_utilization` | `concurrency_utilization`, `requests_per_second`, `cpu_utilization`, `memory_utilization` | | +| `scaling_target` | `100` | 1 to 100 for the utilization metrics, any positive rate for `requests_per_second` | | +| `scaling_buffer` | `0` | >= 0 | Extra replicas above what the metric asks for. Rejected with `cpu_utilization` or `memory_utilization`. | +| `cooldown` | `10` | 0 to 3600 seconds | Time at reduced load before scaling down. | +| `response_grace_period` | `900` | 16 to 43200 seconds | Also the ceiling on an async run, so 12 hours is the maximum. | +| `evaluation_interval_seconds` | `30` | 6 to 300 | Window metrics are evaluated over. | +| `load_balancing_algorithm` | `round-robin` | `round-robin`, `first-available`, `min-connections`, `random-choice-2` | `first-available` suits `replica_concurrency = 1` GPU work. | +| `compute_tier` | `interruptible` | `interruptible`, `protected` | `protected` is on-demand: higher availability, higher price. | +| `roll_out_duration_seconds` | `0` | 0 to 21600 | Gradual traffic shift to a new revision. Keep 0 while iterating. | + +Two combinations the API rejects outright: + +- `cpu_utilization` or `memory_utilization` with `min_replicas = 0`. These metrics need a running + replica to measure, so set `min_replicas` to at least 1. +- `scaling_buffer` with either of those two metrics. + +## Dependencies + +```toml +[cerebrium.dependencies.pip] +torch = "==2.0.0" # exact +transformers = "latest" +numpy = ">=1.26" + +[cerebrium.dependencies.apt] +ffmpeg = "latest" + +[cerebrium.dependencies.conda] +cudatoolkit = "11.7" + +[cerebrium.dependencies.paths] +pip = "requirements.txt" +apt = "pkglist.txt" +``` + +## `[cerebrium.experimental]` + +```toml +[cerebrium.experimental] +checkpointing = true +``` + +Turns on memory and GPU checkpointing (beta, availability varies by account). The container also +has to POST to `http://169.254.169.253:8234/checkpoint` once initialisation is done, so it is a +config change plus a code change. See `references/troubleshooting.md` for where it sits in the +cold-start order. + +## What forces a full rebuild + +Batch these edits together, they are the slow ones: + +- `python_version` +- `docker_base_image_url` +- any `[cerebrium.dependencies.apt]` or `[cerebrium.dependencies.conda]` change + +Pip-only changes and code changes are much cheaper. + +## Worked example + +```toml +[cerebrium.deployment] +name = "llm-inference" +python_version = "3.12" +disable_auth = false +include = ["./*", "main.py", "cerebrium.toml"] +exclude = [".*"] +use_uv = true +deployment_initialization_timeout = 800 + +[cerebrium.hardware] +cpu = 8 +memory = 32.0 +compute = ["HOPPER_H100", "AMPERE_A100_80GB"] +gpu_count = 1 +region = "us-east-1" + +[cerebrium.scaling] +min_replicas = 0 +max_replicas = 10 +replica_concurrency = 1 +scaling_metric = "concurrency_utilization" +scaling_target = 100 +cooldown = 60 +compute_tier = "protected" + +[cerebrium.dependencies.pip] +vllm = "latest" +``` + +## Private base images + +Log in to the registry before deploying an app that pulls one: + +```bash +docker login -u your-dockerhub-username # not the bare `docker login` OAuth flow +``` + +## Checklist before deploying + +- [ ] Every value that matters is in the file, including ones set by a previous `apps scale` +- [ ] `disable_auth` is deliberate, and `false` if the endpoint is not meant to be public +- [ ] `max_replicas` matches the traffic you expect, not the default 1 +- [ ] `replica_concurrency` matches the workload (1 for one-request-per-GPU inference) +- [ ] `compute` uses an accepted identifier, and `cpu`, `memory` and `gpu_count` are inside that + type's limits and the plan's (`references/hardware.md`) +- [ ] weights load from `/persistent-storage`, not baked into the image or in `include` +- [ ] custom runtime: the port in `entrypoint` equals `port` +- [ ] secrets are added with `cerebrium secrets add`, never hardcoded From 59791f4aa55e7d839085c6b765b5b690b47a156c Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" <109931778+mintlify[bot]@users.noreply.github.com> Date: Mon, 31 Aug 2026 18:06:37 +0000 Subject: [PATCH 2/3] fix: use MDX comment syntax in checkpointing page so it parses --- performance/checkpointing.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index 4eeb2b8b..ed5b14d5 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -135,9 +135,9 @@ The most common checkpoint failure is vLLM's default `fork` start method. Set `V The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). - +link customers are told to follow. */} ## When a checkpoint fails From fd210e004ab47cfcdb5f9b2c792d954f25ce5128 Mon Sep 17 00:00:00 2001 From: "mintlify[bot]" Date: Mon, 31 Aug 2026 18:06:56 +0000 Subject: [PATCH 3/3] Prettified Code! --- performance/checkpointing.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/performance/checkpointing.mdx b/performance/checkpointing.mdx index ed5b14d5..4900a056 100644 --- a/performance/checkpointing.mdx +++ b/performance/checkpointing.mdx @@ -135,9 +135,9 @@ The most common checkpoint failure is vLLM's default `fork` start method. Set `V The larger the size of the memory checkpoint, the slower the restore is. Reduce the size of the snapshot substantially and improve startup times by dropping the KV cache before checkpoint and recreating it after restore. vLLM has functionality that does this built in as part of [vLLM Sleep Mode](https://docs.vllm.ai/en/latest/features/sleep_mode/). -{/* The headings below are linked to by anchor from the checkpoint agent's +{/_ The headings below are linked to by anchor from the checkpoint agent's error messages (internal/agent/checkpoint_error.go). Renaming one breaks the -link customers are told to follow. */} +link customers are told to follow. _/} ## When a checkpoint fails