Repository navigation
Conversation
haarchri
reviewed
Oct 20, 2026
| clusters it didn't provision. The agent would report in the status of two custom | ||
| resources. The InferenceCluster would create a PoolObservation for each of its | ||
| pools, and publish the pool's observed node count and histogram from its status. | ||
| It would publish neither for a pool the agent hasn't reported on, or reports it |
Collaborator
There was a problem hiding this comment.
covers the case where the agent knows it is blind. It doesnot cover an agent that crashed or lost its connection, where the last status just stays put. Could PoolObservation carry an observedAt timestamp? The scheduler could then fall back to whole-node accounting when the timestamp is older than N. ?
| can't observe. Each ModelReplica would create a ReplicaObservation for its pods, | ||
| and set its `PodsScheduled` condition to true once the agent reports all of the | ||
| replica's pods scheduled. Provider-kubernetes would watch both resources, and | ||
| copy their status to the control plane as it changes rather than on a poll. |
Collaborator
There was a problem hiding this comment.
provider-kubernetes is running with --enable-watches since the latest version bump #513
| time, putting each on a node with enough free devices. Before it placed a new | ||
| replica, it would: | ||
|
|
||
| 1. Start from the published histogram. The agent would leave out cordoned and |
Collaborator
There was a problem hiding this comment.
observedNodes counts nodes “ready or not”, and step 1 drops unready nodes. So a node that is still booting counts as neither free nor growable, and the pool looks full until the node is ready. That may be fine ... L58 vs. L80 ?!
|
|
||
| 1. Start from the published histogram. The agent would leave out cordoned and | ||
| unready nodes, and treat devices that other workloads claim as taken. | ||
| 2. Add the nodes the pool can still grow by, `nodes` minus `observedNodes`, |
Collaborator
There was a problem hiding this comment.
nodes - observedNodes can go negative during surge upgrades on the provider side?!
|
|
||
| The pool and replica observations would update independently, so a replica | ||
| could report that its pods have scheduled before the histogram counts their | ||
| GPUs. The scheduler would keep placing a replica's pods in the third step for a |
Collaborator
There was a problem hiding this comment.
Composition functions are stateless, so this would need PodsScheduled.lastTransitionTime plus a requeue to expire it. Simpler and exact: keep a replica in step 3 until its lastTransitionTime is older than the pools observedAt. Then we know the histogram already counts it, with no guessing and no double counting ?!
| resources. The InferenceCluster would create a PoolObservation for each of its | ||
| pools, and publish the pool's observed node count and histogram from its status. | ||
| It would publish neither for a pool the agent hasn't reported on, or reports it | ||
| can't observe. Each ModelReplica would create a ReplicaObservation for its pods, |
Collaborator
There was a problem hiding this comment.
How does the agent find a replicas pods? A label selector on the ReplicaObservation? Please also say what all pods scheduled means for Grove and multi-node gangs, where some pods claim nothing.
| nodes with the same number free of each kind. The fleet scheduler would fit | ||
| replicas against these histograms instead of against node counts. | ||
|
|
||
| Modelplane would install the agent as part of the serving stack, including on |
Collaborator
There was a problem hiding this comment.
On BYO clusters, the agent needs cluster-wide read access to Nodes, ResourceSlices and ResourceClaims. Please note the RBAC it needs.
|
|
||
| ### Publish every node's free devices | ||
|
|
||
| The InferenceCluster could publish each node's free devices. A pool's status |
Collaborator
There was a problem hiding this comment.
Nit: since the histograms size depends on device counts (for example 8 GPUs × 32 EFA on p5), please give nodesByFreeDevices a maxItems in the schema sketch, like devices[] has.
There was a problem hiding this comment.
🔵 Needs a closer look
The multinode reservation issue remains unresolved in a broad scheduling and infrastructure change.
2 open issues
What changed in this PR
Implements GPU-aware scheduling using per-node free-device histograms, with cluster-agent observations, schema updates, and documentation.
Changes:
- Adds pool and replica observation resources.
- Updates scheduling to account for free GPUs.
- Installs the cluster agent in serving stacks.
- Updates schemas, tests, and documentation.
| File | Summary |
|---|---|
schemas/python/models/ai/modelplane/inferencecluster/v1alpha1.py |
Adds free-device histogram models. |
schemas/.lock.json |
Updates schema lock data. |
functions/compose-serving-stack/tests/test_fn.py |
Updates serving-stack fixtures. |
functions/compose-serving-stack/function/stacks/common.py |
Installs the cluster-agent chart. Moderate finding (4 votes): update the PR description/status to reflect the implementation. |
functions/compose-model-replica/tests/test_observation.py |
Tests replica observations. |
functions/compose-model-replica/tests/test_fn.py |
Updates replica composition tests. |
functions/compose-model-replica/function/observation.py |
Handles replica observations and scheduling status. |
functions/compose-model-replica/function/fn.py |
Integrates replica scheduling conditions. |
functions/compose-model-replica/function/backends/base.py |
Exposes replica-index labels. |
functions/compose-model-deployment/tests/test_scheduling.py |
Tests histogram-based scheduling. |
functions/compose-model-deployment/function/scheduling.py |
Implements histogram-based placement. Moderate finding (2 votes): preserve distinct-node reservations for multinode gangs. |
functions/compose-model-deployment/function/fn.py |
Supplies scheduling time input. |
functions/compose-inference-cluster/tests/test_fn.py |
Tests pool observation behavior. |
functions/compose-inference-cluster/function/fn.py |
Publishes pool observations. |
docs/content/platform/inference-cluster.md |
Documents agent pool labels. |
docs/content/overview/how-it-works.md |
Updates the scheduling overview. |
docs/content/architecture/scheduling.md |
Documents histogram accounting. |
design/free-gpu-scheduling.md |
Adds the scheduling design. |
apis/inferenceclusters/definition.yaml |
Extends the InferenceCluster status schema. |
Review effort: Lite
Comment on lines
+910
to
+912
| capacity = trial[pool_name].copy() | ||
| if all(capacity.place(_claim(p.device_requests), m.pods) for m, p in zip(engine.members, members, strict=True)): | ||
| trial[pool_name] = capacity |
Comment on lines
+319
to
+325
| Chart( | ||
| key="cluster-agent", | ||
| release="mp-cluster-agent", | ||
| namespace="modelplane-system", | ||
| chart="cluster-agent", | ||
| repository=_CLUSTER_AGENT_REPO, | ||
| version=_CLUSTER_AGENT_VERSION, |
The fleet scheduler charges a whole node for every pod that claims a GPU, so a pod that claims one GPU of a four-GPU node leaves the other three idle. This design proposes fitting pods against a per-pool histogram of nodes by free GPUs, reported by an agent on each InferenceCluster. Towards modelplaneai#172. Signed-off-by: Nic Cope <nicc@rk0n.org>
The fleet scheduler can only see what an InferenceCluster declares about its pools, not which of their devices are free. The cluster agent observes that on the cluster and reports it in the status of PoolObservations and ReplicaObservations. This commit adds the agent's chart to the serving stack's common components, so every cluster gets it, including those Modelplane didn't provision. The chart and its image are expected at xpkg.upbound.io/modelplane/cluster-agent, tagged 0.1.0 and v0.1.0. Signed-off-by: Nic Cope <nicc@rk0n.org>
An InferenceCluster publishes the most nodes each of its pools can have, which is all the fleet scheduler knows about a pool's capacity. This commit adds the nodes a pool has now and a histogram of its schedulable nodes by free devices, which the cluster agent reports. Signed-off-by: Nic Cope <nicc@rk0n.org>
Signed-off-by: Nic Cope <nicc@rk0n.org>
The fleet scheduler reads an InferenceCluster's pools from its status, which only says how many nodes each pool can have. This commit has the InferenceCluster create a PoolObservation on the cluster for each of its pools, once the serving stack has installed the agent, and publish the nodes the pool has and its histogram of nodes by free devices. provider-kubernetes watches each PoolObservation, so the agent's reports reach the InferenceCluster when they change. A pool the agent hasn't reported on, or can't observe, publishes neither, which tells the scheduler to fall back to whole-node accounting for it. Signed-off-by: Nic Cope <nicc@rk0n.org>
The fleet scheduler will fit replicas against each pool's histogram of free devices, which counts the devices a replica's pods hold only once they've scheduled. It needs to know which replicas that is, so it can count the rest against the pool itself. This commit has each ModelReplica create a ReplicaObservation for its engine pods, selected by the deployment and replica index they carry, and relay what the cluster agent reports as a PodsScheduled condition. It does so only once its cluster publishes an observation of one of the replica's pools. The agent installs the ReplicaObservation CRD, and an Object of a kind the cluster doesn't serve can't be deleted, so it would hold up the replica's deletion. The agent's own Scheduled condition is true once every pod it can see is bound, which a gang can satisfy before its workers exist, so the replica also requires as many pods as its engines run. Signed-off-by: Nic Cope <nicc@rk0n.org>
The fleet scheduler charged a whole node for every pod that claims a GPU, however many of the node's GPUs it claims. A one-GPU replica on a pool of four-GPU nodes left three idle, and a prefill and decode replica that fits on one 8-GPU node needed two. This commit fits each pod against how many of a pool's nodes have each set of free devices. Where the cluster agent reports on a pool, that's its histogram plus the nodes the pool can still grow by. The histogram already counts the devices of scheduled pods, so the ledger charges only replicas whose pods haven't scheduled, or scheduled less than a minute ago, since the replica and the histogram update independently. Where the agent doesn't report on a pool, every node starts empty and a pod that claims any device takes a whole one, as before. The scheduler can't know which node the cluster's scheduler will bind a pod to. It assumes the one with the most of the claimed devices free, and a wrong guess leaves a pod Pending until devices free up. Fixes modelplaneai#172. Signed-off-by: Nic Cope <nicc@rk0n.org>
The architecture and overview pages said the scheduler counts capacity in whole nodes, and listed charging a whole node per pod as a known limitation. This commit describes counting the free devices the cluster agent reports instead, including the fallback to whole nodes for a pool it doesn't report on, and notes that an Existing cluster's pool label is how the agent finds a pool's nodes. Signed-off-by: Nic Cope <nicc@rk0n.org>
negz
force-pushed
the
bin-there-done-that
branch
from
October 20, 2026 20:06
71a2e1d to
a3dbd92
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
Fixes #172.
The fleet scheduler charges a whole node for every pod that claims a GPU, however many of the node's GPUs it claims. A Qwen3-8B replica on a pool of four-L4 nodes leaves three GPUs idle, and a prefill and decode replica that fits on one 8-GPU node charges two nodes.
This PR proposes a cluster agent that reports a histogram of each pool's nodes by free devices, and a fleet scheduler that fits replicas against those histograms instead of against node counts. The design is in
design/free-gpu-scheduling.md. I'm opening it as a draft with only the design so we can agree on the direction, and I'll push the implementation to this PR once we have.Agent repo is at https://github.com/modelplaneai/cluster-agent.
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.Added or updated tests covering any composition function changes.git commit -s.