Repository navigation
Built-in metric mappings merge histograms whose buckets don't match #523
Description
What happened?
The built-in mappings rename histograms from different engines, engine versions and deployments onto one modelplane_* name, but nothing guarantees those sources share bucket boundaries. When they don't, a fleet quantile over the shared name is wrong while looking plausible.
The module docstring (metrics.py#L21-L27) and the MetricMapping XRD both say a histogram is only renamed onto a shared name where its buckets match. That holds when a histogram's buckets are fixed per component, as they are for the gateway and the endpoint picker. It doesn't hold for these:
- vLLM and SGLang token counts. metrics.py#L50-L51 and #L72-L73 both feed
modelplane_request_input_tokensandmodelplane_request_output_tokens. vLLM v0.23.0 buckets them 1, 2, 5, 10, 20, 50, 100, 200, 500, …; SGLang v0.5.12.post1 uses 100, 300, 500, 700, 1000, 1500, … - vLLM token counts across deployments. vLLM builds both token histograms with
build_1_2_5_buckets(max_model_len), so the top bucket follows each deployment's--max-model-len. The examples use 8192, 16384, 40960 and 131072, which gives top buckets from 5000 to 100000. Two vLLM deployments on the same version are enough to break the quantile. - SGLang token counts across versions and flags. The default list changed between v0.5.10.post1 (…, 12000, 15000, 20000, 22000, …, 66000, 99000, 132000, …) and v0.5.12.post1 (…, 12500, 15000, 17500, …, 60000, 80000, 100000, 200000, …). The repo pins both.
--prompt-tokens-bucketsand--generation-tokens-bucketsoverride it per deployment. - vLLM latency across versions. v0.7.3 runs the V0 engine by default. Its TTFT buckets stop at 10.0 and its e2e and queue buckets stop at 60.0. v0.9.2 and later extend them to 2560 and 7680. The llama-3.1-8b recipe pins v0.7.3, and the others pin v0.11.0 or later.
Run through the collector, a mismatch shows up two ways. Where one source has a bucket the other lacks, sum by (le) stops being monotonic. Where one source's top bucket is lower, its observations above that bucket only count toward +Inf, so the fleet histogram puts them above the other source's highest bucket. Both happen in the reproduction below. In each case every engine serves the same 1000 requests:
# Case 1: vLLM v0.23.0 and SGLang v0.5.12.post1, every prompt 150 tokens
sum by (le) (modelplane_request_input_tokens_bucket)
le=100 0 le=200 1000 le=300 1000 le=500 2000 le=700 1000 le=1000 2000
histogram_quantile(0.9, sum by (le) (...)) 460 (true: 150)
by (deployment, le) vLLM 190, SGLang 280
# Case 2: vLLM v0.23.0, max_model_len 4096 and 32768, every prompt 3000 tokens
sum by (le) (modelplane_request_input_tokens_bucket)
le=2000 0 le=5000 1000 le=10000 1000 le=20000 1000 le=+Inf 2000
histogram_quantile(0.5, sum by (le) (...)) 5000 (true: 3000)
histogram_quantile(0.9, sum by (le) (...)) 20000
In case 2 the fleet histogram says half the requests had prompts over 20,000 tokens, but the longest prompt was 3000.
Grouping by deployment avoids the cross-deployment cases but not the cross-version one during a rolling engine upgrade, and nothing on a series says which engine or version produced it: engine holds the engine's name from the ModelReplica, not vLLM or SGLang.
How can we reproduce it?
- Render the collector config with
collector.config(...)from main,metrics.BUILTIN_MAPPINGSand aprometheussink. Replace the receiver's scrape configs with static targets labeled withdeployment,replica,engineandnamespace. - Serve a static
/metricsfrom each target with the source histograms above, using each engine's real bucket list and 1000 identical observations. - Run
otel/opentelemetry-collector-contrib:0.161.0with that config, and have Prometheus v3.5.0 scrape its exporter. - Query
histogram_quantile(0.9, sum by (le) (modelplane_request_input_tokens_bucket)).
What environment did it happen in?
Modelplane version: main at 0ee6ab4 (also v0.5.0)
Collector: otel/opentelemetry-collector-contrib:0.161.0
Engines: bucket layouts from vLLM v0.7.3, v0.11.0 and v0.23.0, and SGLang v0.5.10.post1 and v0.5.12.post1, the versions the repo pins
Backend: Prometheus v3.5.0
Activity
dennis-upbound commentedon Oct 20, 2026
Options for this, before anyone builds one. Only
modelplane_request_input_tokensandmodelplane_request_output_tokenstake histograms from two engines. The latency histograms (ttft,duration,queue,prefill,decode) come from vLLM alone, so the mismatch there is only between vLLM versions.sum by (deployment, le), neversum by (le). A deployment's engines are one kind of engine and normally share a--max-model-len, which fixes cases 1 and 2. Rolling a deployment to a new engine version still mixes buckets for the length of the roll. The telemetry docs would say this, beside each histogram.modelplane_vllm_request_input_tokens. This keeps vLLM and SGLang apart even undersum by (le), but vLLM deployments with different--max-model-lenstill don't share buckets. It also breaks any dashboard that compares engines. Comparing engines is why the names are shared.__meta_kubernetes_pod_container_imageinto animagelabel. Every engine series then says which engine and version produced it. Grouping by(deployment, image, le)then also separates a rolling upgrade's two versions. It's one relabel rule, and adds one label with a few values per deployment.I'd do 1 and 3. Option 2 costs the cross-engine comparison and doesn't fix the per-deployment case that 1 already handles. @negz, does an
imagelabel on engine series fit themodelplane_*metrics?Map SGLang's prefix cache and retraction counters
dennis-upbound commentedon Oct 20, 2026
Replacing the options in my comment above with a direction, since we want this fixed for v0.6.
Those options left a shared name with more than one bucket layout and asked whoever writes the query to group around it. A fleet dashboard is written once, by someone who hasn't read that rule, and
sum by (le)gives a wrong number with only a Prometheus info annotation to say so. I ran the repro into Prometheus v3.5 and v3.15, classic and NHCB: no setting gets the right answer, and v3.15's NHCB bucket reconciliation still gives p90 = 460 against a true 150. A central Prometheus doesn't change that either, since the repro already used one.The rule I'd adopt: every
modelplane_*histogram has exactly one bucket layout across the fleet, so the obvious query is the right one.That already holds for the gateway and the picker. It holds for the engine latency histograms once nothing runs vLLM older than v0.9.2: their buckets are identical from v0.9.2 to v0.27.1, and only the llama-3.1-8b recipe still pins v0.7.3. It can't hold for the token histograms. vLLM sizes them from
--max-model-len, so two vLLM deployments disagree, and SGLang's change between releases and with--prompt-tokens-buckets. Per-engine names wouldn't help, because vLLM disagrees with itself.For v0.6:
part: Sumandpart: Count, ontomodelplane_engine_tokens_total{direction}andmodelplane_engine_requests_total, and stop exporting the buckets undermodelplane_request_input_tokensandmodelplane_request_output_tokens. I ran this through the collector against vLLM and SGLang bucket layouts and against two vLLM context lengths. Totals per deployment and across the fleet were exact, and so was the fleet mean prompt length (1575, the true value), with no bucket series left. Theengine_prefix keeps these apart from the gateway'smodelplane_tokens_totalin the design, so the two can't be added together by accident.sglang:queue_time_secondsout by hand for this reason, and the test means the next mapping doesn't depend on someone remembering.What it costs: the normalized engine metrics no longer carry a token-length distribution, including the per-deployment one, which was correct. Two places can supply it later, and neither blocks this. The gateway's
gen_ai_client_token_usagehas fixed buckets and is planned formodelplane_tokens_totalin #476.passthroughon a MetricMapping (telemetry.md#L225) is in the design but not in the XRD yet. Anyone on v0.5.0 using the buckets ofmodelplane_request_input_tokensloses them, so this needs a release note and an update to the metric surface table (telemetry.md#L621).The engine counters still double when a prefill/decode deployment's two legs are summed. That's #525, which needs a phase label rather than anything here.
An
imagelabel (option 3 above) would still help compare engines and separate a rolling upgrade, but this issue no longer needs it.@negz does this work for you?
Export the engines' token histograms as counters
Check the built-in mappings against scrapes of what they read
Export the engines' token histograms as counters
Check the built-in mappings against scrapes of what they read
Check the built-in mappings against scrapes of what they read
Export the engines' token histograms as counters
Check the built-in mappings against scrapes of what they read
Check the built-in mappings against scrapes of what they read
Check the built-in mappings against scrapes of what they read
Export the engines' token histograms as counters
Check the built-in mappings against scrapes of what they read