Compose the router the user writes for prefill/decode serving by dennis-upbound · Pull Request #529 · modelplaneai/modelplane

dennis-upbound · 2026-10-16T18:18:47Z

Description of your changes

This builds the design in #514. Merge it after #514, since design/design.md links to the design doc there.

Modelplane writes one prefill/decode router. It injects the llm-d sidecar with --kv-connector=nixlv2 and VLLM_NIXL_* environment, and reads --port from the decode engine's args to find the engine behind the sidecar. An SGLang pair gets that router and reports Ready while its decode engine rejects every request (#502). A container written beside the engine is dropped (#108), and the picker's configuration can't be changed (#179).

A PrefillDecode deployment now writes its routing. CEL requires serving.router: EndpointPicker, which requires serving.endpointPicker, and the user writes the sidecar into the decode member. serving.endpointPicker.template is the picker's pod, with one container, epp, whose image and command line the user writes. Each picker spells its flags differently, so Modelplane passes what only it knows as MODELPLANE_POOL_NAME, MODELPLANE_POOL_NAMESPACE and MODELPLANE_PICKER_CONFIG, and fixes only the ports: 9002, 9003 for its readiness probe, and 9090 for metrics. config is optional, for a picker that takes none. Modelplane labels each engine's pods with its role and composes the InferencePool and HTTPRoute. It injects nothing into the user's pods, and warns when the decode pod has no container beside the engine. Unified keeps llm-d's picker and Modelplane's default configuration unless it sets router.

Member templates can hold containers beside engine. Every container takes ports and resources, whose keys are limited to cpu, memory and ephemeral-storage: a GPU there would sit beside the device claim, outside what the scheduler counts. A container beside the engine can also set a readinessProbe, such as the sidecar's /health. The backends pass those containers through as written and decorate only engine, whose resources sit beside its device claim. The engine's port named http, 8000 unless it declares one, is where Modelplane probes it and the collector scrapes it. CEL checks ports at the member, so a Worker is exempt:

At most one container in a serving pod declares 8000.
When another container takes 8000, the engine declares http elsewhere, and when the engine's http is elsewhere, another container takes 8000.
The engine may name 8000 only http, and port names are unique in a container.

Other changes:

The picker gets a gRPC readiness probe, and its Object is watched. Without the probe, a configuration change rolled the picker before the new pod could serve, and the gateway, which fails closed, returned HTTP 500s. A configuration the picker rejects leaves the old picker serving, and the replica stops being Ready once that rollout passes its progress deadline.
The default Unified configuration uses the engine's block size or 64, whichever is larger. llm-d's picker raises a smaller value to 64 (Enforce a minimum blockSizeTokens to prevent EPP OOMs from oversized prefix cache indexer llm-d/llm-d-router#1158).
PrefillDecode pods get only llm-d.ai/role, the one label llm-d's picker reads, so Modelplane leaves the user's app label alone.
The InferencePool no longer sets failureMode: FailOpen. Envoy AI Gateway hard-codes its ext-proc filter to fail closed, so it did nothing.
The Kimi-K2 recipe is removed until someone can run it again: its manifest isn't valid under this API, and its routing never ran. The ModelDeployment page's disaggregated example is the Qwen3-8B vLLM NIXL pair below. The Unified SGLang recipes label their pods llm-d.ai/engine-type: sglang. Without that label the picker reads vLLM's metric names and sees an empty queue.

This breaks existing PrefillDecode deployments. Their replicas get a fatal result, which leaves the running pods and router in place until the deployment adds its routing. Only unit tests cover that path.

On GKE, Qwen3-8B on two L4s, at 401a67f, before rebasing onto main (the rebase changed only test expectations). Each picker below ran from its template, with $(MODELPLANE_POOL_NAME), $(MODELPLANE_POOL_NAMESPACE) and $(MODELPLANE_PICKER_CONFIG) in its args, went Ready on its gRPC probe, and served its metrics to an unauthenticated scrape. The InferencePools left failureMode at its default, FailClose.

SGLang prefill/decode, written as the routing docs' example: all 22 requests went through prefill and then decode (llm_d_epp_disagg_decision_total{decision_type="prefill-decode"}), and the long prompts returned in about 2.6s. The sidecar started with KVConnector: sglang on 8000, and the decode engine served on 8001 behind it.
vLLM prefill/decode with NIXL, the ModelDeployment page's disaggregated example: the 5 long prompts went through prefill, with 5 NIXL transfers of 542 MB in total and no failures. The 8 short ones went to decode alone, as prefix-based-pd-decider intends. The picker loaded the rendered configuration, with blockSizeTokens as the integer 64.
Unified vLLM with no router, so Modelplane's own llm-d picker in the same template form: 7 of 7 requests returned 200, the picker scheduled all 13 requests it saw, and the default configuration rendered blockSizeTokens: 64.

At ea2ad28, before the picker became a template, the same cluster also covered a vLLM decode gang with the sidecar in the Leader (30 of 30 requests), the no-sidecar warning event, and a picker configuration llm-d rejects, which left the old picker serving and the replica not Ready 607 seconds later.

CEL also ran against a live apiserver with the template form: 33 valid and invalid manifests, including a picker container not named epp, two picker containers, and nvidia.com/gpu in resources on the engine, a sidecar and the picker.

Fixes #502
Fixes #108
Fixes #179

I have:

Read and followed Modelplane's contribution process.
Run nix flake check (or ./nix.sh flake check) and made sure it passes.
Added or updated tests covering any composition function changes.
Signed off every commit with git commit -s.

🤖 Generated with Claude Code

github-actions · 2026-10-16T18:18:59Z

Docs preview: https://modelplane-docs-pr-529.vercel.app (ready once the site's Content workflow finishes)

haarchri

made a few comments a few suggestions to think about before making this PR ready:

A pad picker config fails silently on a running deployment: the new picker pod crash-loops, the old one keeps serving, and the replica still reports Ready. The only signal is the pod log. Can we watch the EPP Deployment’s rollout (updatedReplicas vs readyReplicas, or the Progressing condition) and put a condition on the replica? Right now a bad config is only visible in pod logs.
Sidecars cannot be made "production" ready: The generated Container model has no readinessProbe, resources, securityContext or volumeMounts. The old built-in pd-sidecar had a probe... this implementation opens the template to sidecars but keeps the engine-only field set. At least resources and readinessProbe seem needed for a real sidecar.
endpointPicker.image sounds generic, but it is llm-d only?! - and check we have an gRPC probe on port 9003 hardcoded which is llm-d default health port
if needed a sidecar do not get MODELPLANE_SERVED_MODEL_NAME, so $(MODELPLANE_SERVED_MODEL_NAME) in a sidecar arg stays literal text - is this expected?
under PrefillDecode, Modelplane overwrites any app label the user sets. Is that needed now that the pool selects on the serving label?

haarchri · 2026-10-19T17:19:59Z

                              Unified serves prefill and decode on one engine.
                              PrefillDecode splits them across two engines, each marking
                              its phase as Prefill or Decode.
+                          router:


Under Unified, router: EndpointPicker with no endpointPicker gives the same result as leaving router out. So there are two ways to say the same thing. Is that on purpose? If yes, please say it in the description. Right now “the cluster routes the deployment” sounds different, but it is the same picker ?

Not on purpose, thanks. router: EndpointPicker now requires endpointPicker with image and config. Omitting router gets the default, and setting it replaces it. The descriptions and #514 say so.

haarchri · 2026-10-19T17:21:02Z

                                x-kubernetes-validations:
                                - rule: "!has(self.worker) || self.role == 'Worker'"
                                  message: "worker may only be set on a Worker member"
+                                - rule: "(has(self.role) && self.role == 'Worker') || !has(self.template.spec) || self.template.spec.containers.filter(c, has(c.ports) && c.ports.exists(p, p.containerPort == 8000)).size() <= 1"


These four port rules are hard to read, and they are copied 1:1 into the ModelReplica XRD (L178–185). Please add a short comment in the YAML that lists the rules in plain words and says “keep in sync with modelreplicas”.

Also, did you check the CEL cost estimate on a real apiserver with maxItems 8 × 8?

Added a comment above the rules in both XRDs that lists them in plain words and points at the other XRD. On cost: both CRDs, with these rules at maxItems 8 × 8, created on a kind API server, and the 25 CEL cases ran there too.

haarchri · 2026-10-19T17:21:36Z

                      certificate. An InferenceGateway validates the gateway's
                      serving certificate against it. Written once cert-manager on
                      the cluster has issued.
+              routers:


Nothing reads this yet (it waits on #418), and it is always [EndpointPicker]. I would leave it out until #418 needs it, following “a field arrives with the capability that needs it”

Agreed, dropped. #514 now has it arriving with #418.

haarchri · 2026-10-19T17:22:19Z

        for engine in self.xr.spec.engines:
            backend = _BACKENDS[base.select_backend(engine, stack)]()
            composed.update(backend.build(self.xr, engine, pc, label, stack))
+        if routing.prefill_decode_routing_missing(self.xr):


The missing-routing check runs after every backend has already built its workload. Please move it before the loop so we stop early. The result is the same, but it is cleaner.

Moved before the backend loop.

haarchri · 2026-10-19T17:26:22Z

+                            # closed, rejects requests in between. No service
+                            # name: the overall check, which every picker on
+                            # this command line answers.
+                            "readinessProbe": {"grpc": {"port": 9003}, "periodSeconds": 2},


The picker args set --grpc-port=9002 but not the health port, so we depend on the image’s default. Since users can now bring their own picker image, please add a _EPP_GRPC_HEALTH_PORT constant and also pass it as a flag, if the picker supports --grpc-health-port ?

Added _EPP_GRPC_PORT and _EPP_GRPC_HEALTH_PORT. The picker gets --grpc-health-port=9003, which llm-d-router v0.9.0 supports.

haarchri · 2026-10-19T17:27:19Z

    return next(c for c in member.template.spec.containers if c.name == "engine")


+def sidecar_containers(member: v1alpha1.Member) -> list[dict]:


he sidecar is passed through as written. But the container schema only has name/image/command/args/env/ports, with no readinessProbe, resources or volumeMounts. The old built-in pd-sidecar had a readiness probe. Now the decode pod can go Ready before the sidecar on :8000 is listening. Is that OK?

Not OK, you're right. Containers beside the engine can now set readinessProbe (httpGet, tcpSocket or grpc), and every container resources. The docs and the Kimi-K2 recipe probe the sidecar's /health on 8000. CEL rejects a probe on the engine, which Modelplane probes itself.

haarchri · 2026-10-19T17:28:04Z

+# whose metrics port has the same name. Matching the container too keeps the job off a port another
+# container in the pod names http, such as a sidecar the user writes.
 _METRICS_PORT = "http"
+_METRICS_CONTAINERS = "engine|epp"


Small suggestion: build it from the real container names ("engine" and the EPP name in routing.py), so a rename can’t break scraping without anyone noticing?

The collector is in compose-serving-stack, a separate function package, so it can't import compose-model-replica's names. Both sides now say where the other is.

haarchri · 2026-10-19T17:30:28Z

+          config:
+            apiVersion: llm-d.ai/v1alpha1
+            kind: EndpointPickerConfig
+            plugins:


The old built-in P/D config had disagg-headers-handler, which set the header the sidecar uses to find the prefill pod. It is gone here and in routing.md. Is that because llm-d v0.9 merged it into disagg-profile-handler?

Yes. llm-d-router v0.9.0 deprecates disagg-headers-handler, because disagg-profile-handler now implements PreRequest itself, which is what set the header.

haarchri · 2026-10-19T17:31:02Z

+                args:
+                - --secure-proxy=false
+                - --kv-connector=nixlv2
+                - --vllm-port=8001


The decode port 8001 now lives in three places: --vllm-port, --port, and the http port. Nothing checks that they match ? easy to get this wrong ?

Right, and nothing checks it, because Modelplane doesn't read engine args. CEL catches the port declarations disagreeing, but not the flags. The routing docs now say to keep the three the same.

dennis-upbound · 2026-10-19T18:18:27Z

Thanks, these were good. Changes are in bc60ea0, and the inline replies cover the rest:

A rejected picker config: the picker's Object now stops being Ready when its Deployment's rollout passes the progress deadline (Progressing=False), so the replica stops being Ready too. That's 10 minutes, plus provider-kubernetes' poll interval (ModelDeployment and ModelEndpoint report Ready while the engine has no ready pods #497). The reason stays in the new picker pod's log, which the control plane can't read.
Production sidecars: containers beside the engine take readinessProbe, and every container takes resources, merged on the engine with its device claim. volumeMounts and securityContext stay a future improvement in Propose user-authored routing for prefill/decode serving #514.
endpointPicker.image: Modelplane runs any image with llm-d's command line, which the Gateway API Inference Extension's reference picker shares. The health port is now explicit with --grpc-health-port.
$(MODELPLANE_SERVED_MODEL_NAME) in a sidecar: that's expected. Only the engine gets the MODELPLANE_ environment, and the routing docs now say so.
The app label: dropped, along with llm-d.ai/inference-serving. llm-d-router v0.9.0 reads neither, so PrefillDecode pods get only llm-d.ai/role.

dennis-upbound · 2026-10-20T15:54:29Z

@negz @haarchri ready for another look. Since Chris's review: router now requires endpointPicker; containers take resources and a sidecar readinessProbe; status.routers is gone; there's a no-sidecar warning; and the picker's Object is watched, so a rejected picker config takes the replica out of Ready within seconds of the rollout deadline instead of about an hour. All of it ran on GKE at ea2ad28. The results are in the description. The design changes are in #514.

negz

This looks close to ready to me. The main open thing is the picker thread on #514: if the picker becomes a pod template the user writes, _epp_objects and the endpointPicker schema here change with it. It also needs a rebase, since it conflicts with main in routing.py, test_backends.py and the collector tests.

negz · 2026-10-21T19:23:40Z

                                                                type: string
                                                              apiVersion:
                                                                type: string
+                                                resources:


The description says CPU and memory, but requests and limits take any key, so nvidia.com/gpu: 8 passes and Modelplane puts it on the engine beside its DRA claim. Should CEL limit the keys to cpu, memory and ephemeral-storage? Today a member gets GPUs only through nodeSelector, and the scheduler counts capacity from it.

Yes. CEL now limits requests and limits to cpu, memory and ephemeral-storage on every container, the picker's included, with maxProperties: 3 so the rule's cost stays bounded. I checked it against a live apiserver: nvidia.com/gpu is rejected on the engine, a sidecar and the picker.

negz · 2026-10-21T19:23:40Z

+    assert case.want in routing._unified_epp_config_yaml(case.block_size), case.reason
+
+
+PICKER_CASES = [


Could these be cases in the existing entry-point tables instead of new tables for private helpers (here and elsewhere)? A Unified replica that sets router: EndpointPicker has no test_apply case, and no test_compose case covers the decode-without-sidecar warning. CONTRIBUTING.md asks for new behaviour as cases in the entry point's table, comparing the whole output.

Done. The private-helper tables are gone, and their cases are now test_apply and test_compose cases that compare the whole output. That adds Unified with router: EndpointPicker, a picker with no config, a user picker with env, resources and pull secrets, and a config with out-of-order keys and whole floats. test_compose gains the decode-without-sidecar warning and PrefillDecode with a router but no picker.

negz · 2026-10-21T19:23:40Z

+route between them. This recipe serves an INT4 quantization of the model; the
+native FP8 weights need four such nodes.
+
+The engines ran end to end with the `InferenceClass` below. The


Recipes are the validated set, and this page is under "Validated deployments", but the routing it now uses hasn't run. Could we run it before merging, or hold the recipe change until someone can? The inference-class.yaml and inference-cluster.yaml headers also still say "validated serving Kimi-K2 prefill/decode".

We can't run Kimi-K2 (two H200 nodes with EFA), and its old form isn't valid anymore, so I removed the recipe until someone can rerun it. The disaggregated example on the ModelDeployment page is now the Qwen3-8B vLLM NIXL pair, in the new picker form, which I ran on GKE at 401a67f.

negz · 2026-10-21T19:23:41Z

            "selector": {"matchLabels": selector},
            "targetPorts": [{"number": base.ENGINE_PORT}],
-            "endpointPickerRef": {"name": f"{name}-epp", "port": {"number": 9002}, "failureMode": "FailOpen"},
+            "endpointPickerRef": {"name": f"{name}-epp", "port": {"number": _EPP_GRPC_PORT}, "failureMode": "FailOpen"},


Nit: the readiness probe comment says the gateway fails closed, but the pool still sets failureMode: FailOpen. If Envoy AI Gateway ignores it, could we drop it or say so here? Otherwise a reader will assume a picker outage sends requests straight to the pods.

Good catch. AI Gateway v1.1.0 hard-codes FailureModeAllow: false on the ext-proc filter (inferencepool.go:314), so FailOpen did nothing. I dropped it, which leaves the pool at its default, FailClose, and the comment says why.

dennis-upbound · 2026-10-21T21:23:32Z

@negz @haarchri ready for another look. The picker is now a pod template whose command line the user writes, with Modelplane passing the pool and the config path as MODELPLANE_* env and fixing only the ports. resources take only cpu, memory and ephemeral-storage, the helper tables are entry-point cases, FailOpen is gone, and the Kimi-K2 recipe is out until it can run again. All of it ran on GKE at 401a67f: SGLang and vLLM NIXL prefill/decode and the default Unified picker, details in the description.

Modelplane composed one prefill/decode router: it injected the llm-d sidecar with --kv-connector=nixlv2 and VLLM_NIXL_* environment into every disaggregated pod, and read --port from the decode engine's args. An SGLang pair got a router it couldn't use and reported Ready while it rejected requests, and a container beside the engine was silently dropped. The API adds serving.router (one value, EndpointPicker) and serving.endpointPicker (image and config) to ModelDeployment and ModelReplica, and containers beside engine with ports on member templates. CEL requires both under PrefillDecode, exactly one engine container, and at most one container on port 8000. The models are regenerated. A new Configure Routing page and the disaggregated serving section of Deploy a Model document it. PrefillDecode now requires serving.router: EndpointPicker and a picker config, so routing.apply composes only wiring for it: role labels, the InferencePool, the HTTPRoute, and the user's picker image and config, rendered with sorted keys and whole numbers as ints so its checksum holds still. A PrefillDecode replica stored before that rule gets a fatal result instead of a router its engines can't use, which leaves its running pods and router in place until its deployment adds the routing. Unified keeps Modelplane's default picker configuration when no router is set. The backends pass containers beside the engine through as written and decorate only the engine, whose http port sets its probe. The served model name goes on the engine only. The collector's engine job also matches the engine or picker container, so a sidecar that names a port http isn't scraped. The Kimi-K2 recipe writes its routing: router: EndpointPicker, a prefill/decode picker configuration with blockSizeTokens matching its --block-size=64, the llm-d sidecar on the decode member, the engine's http port, and the NIXL address on both engines. It says this form hasn't run live yet. design.md describes the InferencePool and the user-written routing, and its disaggregated example writes it too. The Unified SGLang recipes (Laguna, Qwen3-Coder FP8) now label their pods llm-d.ai/engine-type: sglang. Without it the picker read vLLM's metric names and saw a queue depth of zero for every SGLang pod. The endpoint picker gets a readiness probe on its gRPC health port. Without it, a picker config change rolled the picker before the new pod had synced its pool, and the gateway, which fails closed, returned HTTP 500s. The default Unified picker configuration renders blockSizeTokens as the engine's block size or 64, whichever is larger. llm-d-router raises a smaller value to 64 at request time (llm-d/llm-d-router#1158), so the configuration Modelplane writes now matches what the picker runs. CEL checks container ports at the member, so a Worker is exempt from all of them. Port names must be unique in a container, the engine may name port 8000 only http, and an unnamed 8000 on the engine is its http port rather than a second 8000 entry. A Worker's engine ports pass through as written, and an empty picker config is rejected. router: EndpointPicker requires endpointPicker with both image and config, so omitting router is how a Unified replica gets the default. Containers take resources, merged on the engine with its device claim, and a container beside the engine can set a readinessProbe. A replica whose decode pod has no container beside the engine gets a warning. PrefillDecode pods carry only llm-d.ai/role, since llm-d reads neither app nor llm-d.ai/inference-serving. The picker gets --grpc-health-port, and its Object stops being Ready when its rollout passes the progress deadline, so a rejected configuration shows on the replica. The picker's Object is watched, so that reaches the replica when it happens, not at provider-kubernetes' next resync. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

Modelplane ran the picker image the user named with llm-d's command line, so router: EndpointPicker meant llm-d's picker. That command line changes between llm-d releases, and other pickers spell it differently: the Gateway API Inference Extension's lightweight picker has no --config-file or --metrics-endpoint-auth flag, and would exit on it. serving.endpointPicker now takes a pod template with one container, epp, whose image and args the user writes, as a member's template does for an engine. Modelplane sets MODELPLANE_POOL_NAME, MODELPLANE_POOL_NAMESPACE and, with a config, MODELPLANE_PICKER_CONFIG in its environment, and declares the ports every picker serves: 9002, 9003 for the readiness probe, and 9090 for metrics. config is optional, for a picker that takes none. A Unified replica without router runs llm-d's picker, written the same way. A container's resources take only cpu, memory and ephemeral-storage. A GPU set there sat beside the member's DRA claim, outside what the scheduler counts. The InferencePool no longer sets failureMode: FailOpen. Envoy AI Gateway hard-codes the ext-proc filter to fail closed, so it did nothing. The private-helper test tables are now cases in test_apply and test_compose, which compare the whole output. The Kimi-K2 recipe is removed until it runs again: its manifest isn't valid under this API, and its routing never ran. The ModelDeployment page's disaggregated example is a Qwen3-8B vLLM NIXL pair instead. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

The picker's ports were part of the contract only in the docs: a command line wrote 9002, 9003 and 9090 out, and nothing tied those numbers to the ports Modelplane declares and probes. Modelplane now sets MODELPLANE_GRPC_PORT, MODELPLANE_GRPC_HEALTH_PORT and MODELPLANE_METRICS_PORT in the picker's environment beside the pool and the configuration's path, and the default picker, the examples and the design name them in their args. A later release can change a port without breaking a deployment's picker. The routing design's issue references are links. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

dennis-upbound · 2026-10-21T22:52:22Z

This also picks up @negz's two comments from #514, which is merged: the picker's ports are now in its environment as MODELPLANE_GRPC_PORT, MODELPLANE_GRPC_HEALTH_PORT and MODELPLANE_METRICS_PORT, which the default picker and the examples name in their args (40b87aa), and the design's issue references are links. The ports are expanded the same way as the pool and config variables that ran on GKE; the new variables themselves haven't run live.

negz · 2026-10-21T22:25:06Z

+                            type: object
+                            description: >-
+                              The endpoint picker Modelplane runs in front of the serving pods.
+                              Required exactly when router is EndpointPicker. Modelplane composes the


Suggested change

Required exactly when router is EndpointPicker. Modelplane composes the

Required when router is EndpointPicker. Modelplane composes the

Seemed a bit like filler.

negz · 2026-10-21T22:30:10Z

+  structural shape so fields can be added without restructuring. It carries
+  one container named `engine` (the inference engine) with its image, command,
+  args, env, and ports, plus `imagePullSecrets`. It can carry other containers
+  beside `engine`, such as a routing sidecar, which Modelplane passes through as
+  written; Modelplane decorates only `engine`. The engine's port named `http`
+  is where Modelplane probes it and scrapes its metrics, 8000 by default.
+  Exactly one container in a serving pod serves 8000: a container in front of
+  the engine declares 8000, and the engine then declares its `http` port
+  elsewhere. Secret references are passed through; the referenced objects must
+  exist on every InferenceCluster the deployment may target.


I'm generally not a fan of retroactively updating/extending designs. I'd rather keep the original design for posterity and add new designs or one pagers that extend it. In this case I think your design from #514 covers that.

negz · 2026-10-21T22:55:38Z

@@ -0,0 +1,474 @@
+---
+title: Configure Routing


I think a reader would expect a page called "Configure Routing" to cover the ModelService layer: weighted splits, failover, routing across deployments. That's all on Expose a Model. This page covers routing inside one cluster, and in practice only matters for prefill/decode, since a Unified deployment doesn't write any routing. It's also the longest page under Models at 472 lines, about twice Deploy a Model or Expose a Model, for something many users won't touch.

What about retitling it "Configure Prefill/Decode", and trimming it to roughly the size of its siblings? Some of it applies to every deployment, so I'd move it to Deploy a Model:

Containers beside the engine, with their ports, resources and readinessProbe. A log shipper uses these as much as a routing sidecar does.

The llm-d.ai/engine-type label, which every Unified SGLang deployment needs.

Tuning the Unified picker could become a short section there too.

negz · 2026-10-21T22:55:39Z

+The engines come from the [Laguna SGLang
+recipe](https://github.com/modelplaneai/modelplane/blob/main/docs/manifests/recipes/laguna/model-deployment-sglang.yaml),
+with the disaggregation flags from llm-d's [SGLang
+guide](https://github.com/llm-d/llm-d/tree/v0.9.0/guides/pd-disaggregation/modelserver/gpu/sglang/base).


Could this example move to docs/manifests/concepts/, with the page embedding it through the manifests shortcode like Deploy a Model does? Inline, the docs-manifests check doesn't validate it against the schema, and readers can't kubectl apply it. It could be an SGLang tab next to the vLLM pair in Deploy a Model's examples.

negz · 2026-10-21T22:55:40Z

+    weight: 2
+  - pluginRef: queue-scorer
+    weight: 1
+```


This reads as a migration note for one release ("Before this release…"). I'd put it in the release notes rather than the docs, which should describe how Modelplane works today.

negz · 2026-10-21T22:55:41Z

+    so these pass through untouched. Each member has its own template, so the
+    user chooses which pods of a gang run them.
+    """
+    assert member.template.spec is not None


This is an existing problem, not one this PR introduces, but we missed it until now. Neither XRD requires a member's template.spec, so a deployment with template: {} passes admission and then fails every reconcile with an AssertionError here, and at the same assert in each backend.

Since this PR touches those schemas anyway, could we make spec required on the member template in both XRDs? That matches the picker's template. With the regenerated model these asserts can go, along with the spec is None branches in _inject_served_model_name and decode_without_sidecar and the !has(self.template.spec) guards in the port rules.

negz · 2026-10-21T22:58:50Z

@@ -0,0 +1,29 @@
+{{- /*


Nice idea but this should at least be a separate commit if not a separate PR.

negz · 2026-10-21T23:06:25Z

        if not self.resolve_inputs():
            return
-        self.compose_model_serving()
+        if not self.compose_model_serving():


Won't this leave a stored prefill/decode replica's ModelAccepted and ModelReady as Unknown? On a fatal result Crossplane keeps the conditions a function sets, and marks every custom condition it didn't set as Unknown, with reason FatalError (reconciler.go). So a pair that's still serving would show ModelReady=Unknown.

Do we need an early return here? As far as I can tell derive_conditions only reads observed resources, so it would still report the real state alongside the fatal result. PrefillDecodeRoutingMissing would then expect those conditions too.

Good catch, you're right. Fixed in #555: the function derives both conditions from observed state even when it stops with the fatal result, and PrefillDecodeRoutingMissing now models a stored pair that still serves and expects ModelReady=True beside the fatal result.

negz · 2026-10-21T23:11:02Z

+    return all(c.name == "engine" for c in member.template.spec.containers)
+
+
+def _default_picker() -> dict[str, Any]:


Suggested change

def _default_picker() -> dict[str, Any]:

def _default_unified_picker() -> dict[str, Any]:

Nit: This threw me until I realized it was needed for Unified mode. Might be clearer.

Renamed in #555.

dennis-upbound mentioned this pull request Oct 16, 2026

Propose user-authored routing for prefill/decode serving #514

Merged

2 of 4 tasks

dennis-upbound force-pushed the dennis/pd-disagg branch 2 times, most recently from 540e6e6 to 4074f46 Compare October 16, 2026 18:27

dennis-upbound mentioned this pull request Oct 16, 2026

The first model on an InferenceGateway reports RoutingReady while the gateway restarts and returns HTTP 500 #531

Open

dennis-upbound force-pushed the dennis/pd-disagg branch 3 times, most recently from 8a26f64 to 2179e9a Compare October 16, 2026 19:48

haarchri reviewed Oct 19, 2026

View reviewed changes

dennis-upbound force-pushed the dennis/pd-disagg branch 2 times, most recently from c08d6e1 to bc60ea0 Compare October 19, 2026 18:15

dennis-upbound added this to Modelplane Roadmap Oct 19, 2026

dennis-upbound moved this to In progress in Modelplane Roadmap Oct 19, 2026

dennis-upbound force-pushed the dennis/pd-disagg branch 2 times, most recently from e0469ba to ea2ad28 Compare October 19, 2026 22:27

dennis-upbound requested review from haarchri and negz October 19, 2026 22:27

dennis-upbound mentioned this pull request Oct 19, 2026

ModelDeployment and ModelEndpoint report Ready while the engine has no ready pods #497

Open

dennis-upbound marked this pull request as ready for review October 20, 2026 15:54

negz reviewed Oct 21, 2026

View reviewed changes

dennis-upbound force-pushed the dennis/pd-disagg branch from ea2ad28 to 401a67f Compare October 21, 2026 20:04

dennis-upbound requested a review from negz October 21, 2026 21:23

dennis-upbound force-pushed the dennis/pd-disagg branch from 401a67f to 2eaf1ac Compare October 21, 2026 21:42

dennis-upbound mentioned this pull request Oct 21, 2026

Serving-path components on an InferenceCluster's system nodes run as one pod with no disruption budget, so node maintenance takes a model offline #554

Open

dennis-upbound and others added 3 commits October 21, 2026 15:49

dennis-upbound force-pushed the dennis/pd-disagg branch from 2eaf1ac to 40b87aa Compare October 21, 2026 22:52

negz approved these changes Oct 21, 2026

View reviewed changes

dennis-upbound merged commit 3664d55 into modelplaneai:main Oct 22, 2026
6 checks passed

github-project-automation Bot moved this from In progress to In Review in Modelplane Roadmap Oct 22, 2026

dennis-upbound mentioned this pull request Oct 22, 2026

Report a stored prefill/decode pair's state when its routing is missing #555

Open

4 tasks done

		return next(c for c in member.template.spec.containers if c.name == "engine")


		def sidecar_containers(member: v1alpha1.Member) -> list[dict]:

		assert case.want in routing._unified_epp_config_yaml(case.block_size), case.reason


		PICKER_CASES = [

	Required exactly when router is EndpointPicker. Modelplane composes the
	Required when router is EndpointPicker. Modelplane composes the

		return all(c.name == "engine" for c in member.template.spec.containers)


		def _default_picker() -> dict[str, Any]:

	def _default_picker() -> dict[str, Any]:
	def _default_unified_picker() -> dict[str, Any]:

Conversation

dennis-upbound commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of your changes

Uh oh!

github-actions Bot commented Oct 16, 2026

Uh oh!

haarchri left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennis-upbound commented Oct 19, 2026

Uh oh!

dennis-upbound commented Oct 20, 2026

Uh oh!

negz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennis-upbound commented Oct 21, 2026

Uh oh!

dennis-upbound commented Oct 21, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennis-upbound commented Oct 16, 2026 •

edited

Loading