Skip to content

Install each device stack only where its driver is claimed, adding CPU pools and AMD serving #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
negz wants to merge 4 commits into from
Closed

Install each device stack only where its driver is claimed, adding CPU pools and AMD serving #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
228625e
feat(serving-stack): allow multiple accelerator cpu, nvidia and amd u…
haarchri Oct 22, 2026
7556990
test(e2e): update e2e test to support cpu, nvidia and amd
haarchri Oct 22, 2026
9b41184
feat(recipes): add recipes, update telemetry for amd
haarchri Oct 22, 2026
0d8a59d
feat(docs): update for vale
haarchri Oct 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .vale/styles/config/vocabularies/Modelplane/accept.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,18 @@ vLLM
[Hh]uggingFace
NVIDIA
NVLink
AMD
Instinct
ROCm
amdgpu
CUDA
HBM3e
cpuset
device-metrics-exporter
[Mm]icrojoules
MI325X
CPUs
NUMA
Envoy
Traefik
MetalLB
Expand Down Expand Up @@ -725,6 +737,18 @@ vLLM
[Hh]uggingFace
NVIDIA
NVLink
AMD
Instinct
ROCm
amdgpu
CUDA
HBM3e
cpuset
device-metrics-exporter
[Mm]icrojoules
MI325X
CPUs
NUMA
Envoy
Traefik
MetalLB
Expand Down
5 changes: 4 additions & 1 deletion CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -426,7 +426,10 @@ checks to trip. In order:
file is high-churn upstream. The fork is deleted the day
NVIDIA/aicr#2515 or #2517 lands.
3. Expect `ALLOW`/`DROP` to fail closed on any component the new catalog
adds; classify it with a reason.
adds; classify it with a reason. An `ALLOW` entry also names the DRA
drivers whose device stacks the component belongs to (empty for a
component every cluster needs) - deciding that is part of classifying
it; see design/multi-accelerator.md.
4. Expect the managed-path assertions to fail closed if a values path
moved or a `--set` stopped landing; re-derive the path from the new
chart before loosening anything. `MODELPLANE_VALUES` fails closed too
Expand Down
5 changes: 4 additions & 1 deletion apis/aksclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,10 +128,13 @@ spec:
Determines what workloads this pool runs. System pools
host controllers, gateways, and infrastructure. GPU
pools host inference workloads and get the NVIDIA
driver installed by AKS.
driver installed by AKS. CPU pools host CPU inference
workloads: no GPU driver, and the modelplane.ai/cpu
label and taint.
enum:
- System
- GPU
- CPU
vmSize:
type: string
description: >-
Expand Down
5 changes: 4 additions & 1 deletion apis/civoclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,13 @@ spec:
Determines what workloads this pool runs. System
pools host controllers, gateways, and infrastructure.
GPU pools host inference workloads and are tainted to
exclude non-GPU pods.
exclude non-GPU pods. CPU pools host CPU inference
workloads: a non-GPU size and the modelplane.ai/cpu
label and taint.
enum:
- System
- GPU
- CPU
size:
type: string
description: >-
Expand Down
5 changes: 4 additions & 1 deletion apis/eksclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,13 @@ spec:
Determines what workloads this group runs. System
groups host controllers, gateways, and infrastructure.
GPU groups host inference workloads and use a
GPU-enabled AMI.
GPU-enabled AMI. CPU groups host CPU inference
workloads: a standard AMI, no GPU, and the
modelplane.ai/cpu label and taint.
enum:
- System
- GPU
- CPU
instanceType:
type: string
description: >-
Expand Down
5 changes: 4 additions & 1 deletion apis/gkeclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,10 +124,13 @@ spec:
Determines what workloads this pool runs. System pools
host controllers, gateways, and infrastructure. GPU
pools host inference workloads and are tainted to
exclude non-GPU pods.
exclude non-GPU pods. CPU pools host CPU inference
workloads: a standard node image, no GPU, and the
modelplane.ai/cpu label and taint.
enum:
- System
- GPU
- CPU
machineType:
type: string
description: >-
Expand Down
41 changes: 29 additions & 12 deletions apis/inferenceclasses/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ spec:
enum: [GKE, EKS, AKS, Nebius, Vultr, Civo]
gke:
type: object
required: [machineType, accelerator]
required: [machineType]
properties:
machineType:
type: string
Expand All @@ -68,7 +68,9 @@ spec:
pool. Provisioning input only: the scheduler matches
against spec.devices, not this block, so count here is
the GCP machine's GPU count and need not be restated in
devices.
devices. Omit for a pool with no GPU, which is
composed with role CPU: a standard node image and the
modelplane.ai/cpu label and taint.
required: [type, count]
properties:
type:
Expand All @@ -84,7 +86,7 @@ spec:
maximum: 16
eks:
type: object
required: [instanceType, accelerator]
required: [instanceType]
properties:
instanceType:
type: string
Expand All @@ -103,7 +105,10 @@ spec:
description: >-
GPU accelerator to attach when provisioning the node
group. Provisioning input only: the scheduler matches
against spec.devices, not this block.
against spec.devices, not this block. Omit for a
group with no GPU, which is composed with role CPU: a
standard node image and the modelplane.ai/cpu label
and taint.
required: [type, count]
properties:
type:
Expand All @@ -120,7 +125,7 @@ spec:
maximum: 16
aks:
type: object
required: [vmSize, accelerator]
required: [vmSize]
properties:
vmSize:
type: string
Expand All @@ -139,7 +144,10 @@ spec:
description: >-
GPU accelerator to attach when provisioning the node
pool. Provisioning input only: the scheduler matches
against spec.devices, not this block.
against spec.devices, not this block. Omit for a
pool with no GPU, which is composed with role CPU: a
standard node image and the modelplane.ai/cpu label
and taint.
required: [type, count]
properties:
type:
Expand All @@ -156,7 +164,7 @@ spec:
maximum: 16
nebius:
type: object
required: [platform, preset, accelerator]
required: [platform, preset]
properties:
platform:
type: string
Expand Down Expand Up @@ -197,7 +205,10 @@ spec:
description: >-
GPU accelerator to attach when provisioning the node
group. Provisioning input only: the scheduler matches
against spec.devices, not this block.
against spec.devices, not this block. Omit for a
group with no GPU, which is composed with role CPU: a
standard node image and the modelplane.ai/cpu label
and taint.
required: [type, count]
properties:
type:
Expand All @@ -214,7 +225,7 @@ spec:
maximum: 16
vultr:
type: object
required: [plan, accelerator]
required: [plan]
properties:
plan:
type: string
Expand All @@ -229,7 +240,10 @@ spec:
description: >-
GPU accelerator to attach when provisioning the node
pool. Provisioning input only: the scheduler matches
against spec.devices, not this block.
against spec.devices, not this block. Omit for a
pool with no GPU, which is composed with role CPU: a
standard node image and the modelplane.ai/cpu label
and taint.
required: [type, count]
properties:
type:
Expand All @@ -246,7 +260,7 @@ spec:
maximum: 16
civo:
type: object
required: [size, accelerator]
required: [size]
properties:
size:
type: string
Expand All @@ -261,7 +275,10 @@ spec:
description: >-
GPU accelerator to attach when provisioning the node
pool. Provisioning input only: the scheduler matches
against spec.devices, not this block.
against spec.devices, not this block. Omit for a
pool with no GPU, which is composed with role CPU: a
standard node image and the modelplane.ai/cpu label
and taint.
required: [type, count]
properties:
type:
Expand Down
5 changes: 3 additions & 2 deletions apis/inferenceclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -608,10 +608,11 @@ spec:
PVCs on this cluster. ModelCache reads this to target
the cache PVC.
maxLength: 253
gpuPools:
schedulablePools:
type: array
description: >-
Schedulable GPU node pools on this cluster, derived from the
Schedulable node pools on this cluster - GPU and CPU alike;
every pool whose class declares devices - derived from the
referenced classes and the per-pool node counts.
ModelDeployment scheduling matches against these.
maxItems: 8
Expand Down
12 changes: 6 additions & 6 deletions apis/metricmappings/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,16 +150,16 @@ spec:
as the component wrote it.
fromUnit:
type: string
enum: [Millijoules, Mebibytes, Milliseconds, Nanoseconds, Percent]
enum: [Millijoules, Microjoules, Mebibytes, Milliseconds, Nanoseconds, Percent]
description: >-
What the component measures this in, when that isn't
the unit the name claims. Modelplane converts to the
base unit: millijoules and milliseconds are divided by
a thousand, nanoseconds by a billion, percent by a
hundred, and mebibytes multiplied out to bytes. A
histogram is converted whole - its sum, its bounds and
its bucket boundaries - so its quantiles come out in
the target unit too.
a thousand, microjoules by a million, nanoseconds by a
billion, percent by a hundred, and mebibytes multiplied
out to bytes. A histogram is converted whole - its sum,
its bounds and its bucket boundaries - so its quantiles
come out in the target unit too.

Percent is for a component that counts a saturation
from nought to a hundred where the name says a ratio.
Expand Down
2 changes: 1 addition & 1 deletion apis/modeldeployments/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -453,7 +453,7 @@ spec:
pods: what devices each pod needs from its node.
The scheduler matches it against a candidate
pool's InferenceClass devices (surfaced on
InferenceCluster status.gpuPools) and places the
InferenceCluster status.schedulablePools) and places the
member on a pool that satisfies it, preferring
one pool for the whole engine and splitting
members across pools only when no pool satisfies
Expand Down
5 changes: 4 additions & 1 deletion apis/nebiusclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,10 +101,13 @@ spec:
Determines what workloads this group runs. System
groups host controllers, gateways, and infrastructure.
GPU groups host inference workloads and are tainted to
exclude non-GPU pods.
exclude non-GPU pods. CPU groups host CPU inference
workloads: a CPU platform, no GPU, and the
modelplane.ai/cpu label and taint.
enum:
- System
- GPU
- CPU
platform:
type: string
description: >-
Expand Down
18 changes: 18 additions & 0 deletions apis/servingstacks/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,24 @@ spec:
- Vultr
- Civo
- Existing
draDrivers:
type: array
description: >-
DRA driver names whose device stacks this cluster needs
(gpu.nvidia.com, dra.cpu, ...), collected by the cluster
composition from the claim: DRA devices of the cluster's
referenced InferenceClasses. A component tagged with a
driver installs only when that driver is listed; untagged
components always install. When the field is absent no

@modelplane-bot modelplane-bot Bot Nov 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the absent-means-unfiltered case? compose-inference-cluster always stamps draDrivers, so the None path only serves ServingStacks from before this PR, and AGENTS.md rules out compatibility paths like that. It also gives unset and empty different meanings. What if absent meant no drivers, and the None branches in join and compose-serving-stack's fn.py went along with the "When the field is absent" sentence?

filtering happens and every component installs, so a
ServingStack from before the field existed keeps its
stack until the cluster composition stamps the list.
maxItems: 16
x-kubernetes-list-type: set
items:
type: string
minLength: 1
maxLength: 253
secrets:
type: array
description: >-
Expand Down
5 changes: 4 additions & 1 deletion apis/vultrclusters/definition.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,10 +110,13 @@ spec:
Determines what workloads this pool runs. System
pools host controllers, gateways, and infrastructure.
GPU pools host inference workloads and are tainted to
exclude non-GPU pods.
exclude non-GPU pods. CPU pools host CPU inference
workloads: a non-GPU plan and the modelplane.ai/cpu
label and taint.
enum:
- System
- GPU
- CPU
plan:
type: string
description: >-
Expand Down
32 changes: 25 additions & 7 deletions docs/content/platform/telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ Some examples of the available metrics:

Latency appears twice on purpose. The `frontend_` series are what your caller experienced,
measured at the gateway. The engine's own series are what the engine spent. For
example, if the frontend metric is slow and the engine isn't, you can
example, if the frontend metric is slow and the engine isn't, you can
troubleshoot routing, queueing, or networking issues instead of the model.

Saturation gauges are per replica, so how you combine them decides what you see. Average
Expand Down Expand Up @@ -223,7 +223,7 @@ Anything else the exporter takes goes under `config`, passed through as you wrot
Modelplane doesn't define a schema for an exporter's settings so anything under
the `config` is passed to the collector exactly as written. TLS, retries,
queueing, compression and headers all work and any new settings in the collector
are respected and the sink keeps working.
are respected and the sink keeps working.

To use an authentication scheme Modelplane doesn't compose, define the extension
yourself under `spec.extensions`. Then reference it by its key from the sink's
Expand Down Expand Up @@ -459,12 +459,16 @@ spec:

## GPUs

The GPU series come from a [DCGM
exporter](https://github.com/NVIDIA/dcgm-exporter), not from the engine: a GPU
The GPU series come from a vendor's GPU exporter, not from the engine: a GPU
belongs to a node, and its series have a `node` label rather than a
deployment. The collector discovers the exporter by its pod - an `app` or
`app.kubernetes.io/name` label containing `dcgm`, with a container port named
`metrics` - and scrapes it directly, not through Prometheus.
deployment. On NVIDIA pools that's a [DCGM
exporter](https://github.com/NVIDIA/dcgm-exporter); on AMD pools it's the AMD
GPU operator's
[device-metrics-exporter](https://github.com/ROCm/device-metrics-exporter),
which the serving stack installs with the AMD stack. The collector discovers
either by its pod - an `app` or `app.kubernetes.io/name` label containing
`dcgm` or `metrics-exporter`, with a container port named `metrics` or
`exporter-port` - and scrapes it directly, not through Prometheus.

Where the exporter comes from depends on the cluster:

Expand Down Expand Up @@ -504,6 +508,20 @@ curl -s localhost:9400/metrics | grep -c -E \
'DCGM_FI_DEV_FB_(USED|FREE)|DCGM_FI_PROF_(GR_ENGINE_ACTIVE|PIPE_TENSOR_ACTIVE|DRAM_ACTIVE)|DCGM_FI_DEV_(GPU_TEMP|POWER_USAGE|TOTAL_ENERGY_CONSUMPTION)'
```

On an AMD pool the same `modelplane_*` names come from the
device-metrics-exporter's fields instead. A dashboard built on them needs no
change when a deployment moves between vendors' pools.

| AMD field | Becomes | Unit |
| --- | --- | --- |
| `gpu_used_vram` | `modelplane_gpu_memory_used_bytes` | MiB, converted to bytes |
| `gpu_free_vram` | `modelplane_gpu_memory_free_bytes` | MiB, converted to bytes |
| `gpu_gfx_activity` | `modelplane_gpu_compute_active_ratio` | percent, converted to a ratio |
| `gpu_umc_activity` | `modelplane_gpu_memory_bandwidth_ratio` | percent, converted to a ratio |
| `gpu_junction_temperature` | `modelplane_gpu_temperature_celsius` | Celsius |
| `gpu_power_usage` | `modelplane_gpu_power_watts` | watts |
| `gpu_energy_consumed` | `modelplane_energy_joules_total` | microjoules, converted to joules |

If fields are missing, extend the exporter's metrics ConfigMap - on a GPU
Operator install that's the `dcgmExporter.config` chart value. The
configuration Modelplane installs on the clouds it provisions publishes each
Expand Down
Loading
Loading