Skip to content

Fix broken links: vendor references/config.md for skill.md #320

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mintlify wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Fix broken links: vendor references/config.md for skill.md #320

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .prettierignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,7 @@ available-hardware.mdx
# A checker there fails that repo's CI on any drift, and Prettier re-wraps this file
# if allowed to touch it. Edit the skill upstream, never here.
/skill.md

# Vendored from skills/cerebrium/references/config.md in CerebriumAI/cerebrium-skills so the
# relative references/config.md link inside skill.md resolves. Edit upstream, never here.
/references/config.md
202 changes: 202 additions & 0 deletions references/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,202 @@
# cerebrium.toml

One file per app, read by `cerebrium deploy` and `cerebrium run`.

Two rules to internalise before editing one:

1. **A key you leave out is set to the default, not left alone.** A deploy replaces the whole app
record. If a previous deploy or `cerebrium apps scale` set `max_replicas = 20` and the TOML
you deploy has no `max_replicas`, the app goes back to 1. Keep every value that matters in the
file.
2. **A key the CLI does not recognise is not rejected locally.** A misspelled `max_replica` does
not fail the deploy in the CLI, it just does nothing there. The whole file is uploaded
verbatim, so the backend sees the key too and is free to reject it. Check spelling against the
tables below rather than trusting a successful deploy. A TOML that does not parse at all does
fail the deploy.

Defaults below are what the API applies when the key is absent. Where a published table
disagrees, prefer these, and set anything that matters explicitly.

## `[cerebrium.deployment]`

| Key | Applied when omitted | Notes |
| --- | --- | --- |
| `name` | required | App name. |
| `python_version` | `3.11` | `3.10`, `3.11`, `3.12` and `3.13` are accepted. Changing it forces a full rebuild. |
| `disable_auth` | `true` | **`true` means the endpoint is public.** Set `false` for anything real. |
| `include` | `["./*", "main.py", "cerebrium.toml"]` | Keep model weights out of it. |
| `exclude` | `[".*"]` | |
| `shell_commands` | `[]` | Run at the end of the build. |
| `pre_build_commands` | `[]` | Run before dependencies install. |
| `docker_base_image_url` | `debian:bookworm-slim` | Changing it forces a full rebuild. |
| `use_uv` | `false` | uv instead of pip. Much faster on large dependency trees. |
| `deployment_initialization_timeout` | `600` | Seconds, 60 to 830. Raise it when weights load slowly. |

`cerebrium init` scaffolds `disable_auth = true`. That is a convenience for a first curl, not a
production default. Treat flipping it to `false` as part of the first real deploy.

## `[cerebrium.runtime]`

| Key | Applied when omitted | Notes |
| --- | --- | --- |
| `container_runtime` | platform default | `"v1"` is runc, `"v2"` is gvisor. Any other value fails the deploy locally. |

The sub-tables in the next two sections sit under the same heading and can be set alongside
`container_runtime`.

## `[cerebrium.runtime.custom]`

Only for a custom web server (FastAPI, ASGI, WebSockets, custom batching) or a Dockerfile build.
Omit the section to use the default Cortex runtime.

| Key | Applied when omitted | Notes |
| --- | --- | --- |
| `port` | `8000` | Must match the port inside `entrypoint`. |
| `entrypoint` | required | A list (`["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]`) or a single string. The list form is unambiguous, prefer it. |
| `healthcheck_endpoint` | `""` (TCP ping) | Non-200 marks the instance unhealthy and restarts it. |
| `readycheck_endpoint` | `""` (TCP ping) | Non-200 removes the instance from routing. |
| `dockerfile_path` | unset | Build from a Dockerfile instead. The path must exist locally, and the image then owns its own dependencies and Python version. |

## `[cerebrium.runtime.deepgram]` and `[cerebrium.runtime.rime]`

Partner service runtimes. The table name is the partner, and an app uses one of them or neither.
Omit both unless you are deploying that partner's image.

| Key | Applied when omitted | Notes |
| --- | --- | --- |
| `port` | partner default | Port the partner service listens on. |
| `model_name` | partner default | Partner model identifier, for example `arcana` or `mist`. |
| `language` | partner default | Language code, for example `en`. |

```toml
[cerebrium.runtime.deepgram]
model_name = "arcana"
language = "en"
```

## `[cerebrium.hardware]`

| Key | Applied when omitted | Accepted |
| --- | --- | --- |
| `cpu` | `2.0` | Depends on the compute type and the plan. See `references/hardware.md`. |
| `memory` | `4.0` | GB of host RAM, not VRAM. Same limits apply. |
| `compute` | `"CPU"` | One identifier, or a preference-ordered list of up to 5 from the same family. |
| `gpu_count` | `1` when `compute` is an accelerator, `0` for `CPU` | Per-type maximum, capped again by the plan. |
| `provider` | platform picks | `aws`, `crusoe` or `nebius`. |
| `region` | platform picks | See `references/hardware.md`. |

## `[cerebrium.scaling]`

| Key | Applied when omitted | Accepted | Notes |
| --- | --- | --- | --- |
| `min_replicas` | `0` | 0 to 2000, capped by plan | Above 0 keeps warm capacity and bills for it. |
| `max_replicas` | `1` | 1 to 2000, capped by plan | The most common cause of unexplained queueing. Raise it before load. |
| `replica_concurrency` | `1` on an accelerator, `100` on CPU | >= 1 | In-flight requests per replica. |
| `scaling_metric` | `concurrency_utilization` | `concurrency_utilization`, `requests_per_second`, `cpu_utilization`, `memory_utilization` | |
| `scaling_target` | `100` | 1 to 100 for the utilization metrics, any positive rate for `requests_per_second` | |
| `scaling_buffer` | `0` | >= 0 | Extra replicas above what the metric asks for. Rejected with `cpu_utilization` or `memory_utilization`. |
| `cooldown` | `10` | 0 to 3600 seconds | Time at reduced load before scaling down. |
| `response_grace_period` | `900` | 16 to 43200 seconds | Also the ceiling on an async run, so 12 hours is the maximum. |
| `evaluation_interval_seconds` | `30` | 6 to 300 | Window metrics are evaluated over. |
| `load_balancing_algorithm` | `round-robin` | `round-robin`, `first-available`, `min-connections`, `random-choice-2` | `first-available` suits `replica_concurrency = 1` GPU work. |
| `compute_tier` | `interruptible` | `interruptible`, `protected` | `protected` is on-demand: higher availability, higher price. |
| `roll_out_duration_seconds` | `0` | 0 to 21600 | Gradual traffic shift to a new revision. Keep 0 while iterating. |

Two combinations the API rejects outright:

- `cpu_utilization` or `memory_utilization` with `min_replicas = 0`. These metrics need a running
replica to measure, so set `min_replicas` to at least 1.
- `scaling_buffer` with either of those two metrics.

## Dependencies

```toml
[cerebrium.dependencies.pip]
torch = "==2.0.0" # exact
transformers = "latest"
numpy = ">=1.26"

[cerebrium.dependencies.apt]
ffmpeg = "latest"

[cerebrium.dependencies.conda]
cudatoolkit = "11.7"

[cerebrium.dependencies.paths]
pip = "requirements.txt"
apt = "pkglist.txt"
```

## `[cerebrium.experimental]`

```toml
[cerebrium.experimental]
checkpointing = true
```

Turns on memory and GPU checkpointing (beta, availability varies by account). The container also
has to POST to `http://169.254.169.253:8234/checkpoint` once initialisation is done, so it is a
config change plus a code change. See `references/troubleshooting.md` for where it sits in the
cold-start order.

## What forces a full rebuild

Batch these edits together, they are the slow ones:

- `python_version`
- `docker_base_image_url`
- any `[cerebrium.dependencies.apt]` or `[cerebrium.dependencies.conda]` change

Pip-only changes and code changes are much cheaper.

## Worked example

```toml
[cerebrium.deployment]
name = "llm-inference"
python_version = "3.12"
disable_auth = false
include = ["./*", "main.py", "cerebrium.toml"]
exclude = [".*"]
use_uv = true
deployment_initialization_timeout = 800

[cerebrium.hardware]
cpu = 8
memory = 32.0
compute = ["HOPPER_H100", "AMPERE_A100_80GB"]
gpu_count = 1
region = "us-east-1"

[cerebrium.scaling]
min_replicas = 0
max_replicas = 10
replica_concurrency = 1
scaling_metric = "concurrency_utilization"
scaling_target = 100
cooldown = 60
compute_tier = "protected"

[cerebrium.dependencies.pip]
vllm = "latest"
```

## Private base images

Log in to the registry before deploying an app that pulls one:

```bash
docker login -u your-dockerhub-username # not the bare `docker login` OAuth flow
```

## Checklist before deploying

- [ ] Every value that matters is in the file, including ones set by a previous `apps scale`
- [ ] `disable_auth` is deliberate, and `false` if the endpoint is not meant to be public
- [ ] `max_replicas` matches the traffic you expect, not the default 1
- [ ] `replica_concurrency` matches the workload (1 for one-request-per-GPU inference)
- [ ] `compute` uses an accepted identifier, and `cpu`, `memory` and `gpu_count` are inside that
type's limits and the plan's (`references/hardware.md`)
- [ ] weights load from `/persistent-storage`, not baked into the image or in `include`
- [ ] custom runtime: the port in `entrypoint` equals `port`
- [ ] secrets are added with `cerebrium secrets add`, never hardcoded
Loading