Repository navigation
Serving-path components on an InferenceCluster's system nodes run as one pod with no disruption budget, so node maintenance takes a model offline #554
Description
What happened
On a provisioned Nebius InferenceCluster, the cloud's node-maintenance controller drained and restarted both system nodes, one after the other (NodeDraining → NodeDrained → InstanceStopping → InstanceStopped → InstanceStarting → InstanceStarted → NodeUncordoned, about 7 minutes in all). It cordoned both nodes before draining the first.
The model's GPU engine pods weren't touched: 0 restarts, Ready throughout. But the deployment's endpoint picker runs on the system nodes as a single pod. It was evicted twice, and its replacement stayed Pending for about 4 minutes:
0/4 nodes are available: 2 node(s) were unschedulable, 2 node(s) had untolerated taint(s)
For those minutes the gateway answered every request for the model with 503 upstream connect error or disconnect/reset before headers.
Why
Modelplane places the serving stack on the cluster's system nodes, and most of what's on the request path there runs as one replica, with no PodDisruptionBudget and no spread across nodes:
| Component | On the request path | Replicas | PDB |
|---|---|---|---|
| Endpoint picker (one per ModelReplica) | yes | 1 | none |
Cluster gateway proxy (envoy-modelplane-system-cluster-gateway-*) |
yes | 1 | none |
| CoreDNS (installed by the cloud) | yes | 2, both on one node | minAvailable 1 |
| Modelplane collector | no (metrics gaps) | 1 | none |
| trust-manager, cert-manager, Envoy Gateway and AI Gateway controllers, LWS controller | no (control loops) | 1 each | none |
The fleet InferenceGateway's proxy already handles this: compose-inference-gateway runs several replicas with envoyPDB: {maxUnavailable: 1} and topology spread across hosts and zones. The per-cluster components don't.
Node maintenance is routine on every cloud, so this happens without anyone changing the cluster. The system pool is small (on Nebius by default, 1 to 2 nodes of 4 vCPU and 16 GB), and the cloud can cordon all of it at once.
What I expected
Draining any one system node, or cordoning all of them, doesn't interrupt serving while the GPU engines are healthy.
Related
- InferenceCluster: Let clusters declare their own system node pools, or none at all #550 lets a cluster declare its system pool's size, which spreading needs.
- Compose the router the user writes for prefill/decode serving #529 composes the endpoint picker from a user-written router, still at
replicas: 1.
Activity
dennis-upbound commentedon Oct 21, 2026
Possible fixes, per component, roughly in order of impact:
maxUnavailable: 1, and a spread across nodes. With Compose the router the user writes for prefill/decode serving #529 this could beserving.endpointPicker.replicas, defaulting to 2. On failover the standby starts with a cold prefix-cache view, which approximate routing rebuilds within minutes.EnvoyProxysettings (replicas,envoyPDB, topology spread) for the per-cluster gateway. Small, since the pattern already exists incompose-inference-gateway.Separately, a drain of a GPU node takes a single-replica deployment down for an engine restart. That's a different problem (more replicas, or a maintenance window on the GPU node group where the cloud supports one), so it probably wants its own issue.