Repository navigation
Map SGLang's prefix cache and retraction counters - #544
Merged
dennis-upbound merged 1 commit intoOct 20, 2026
Merged
dennis-upbound merged 1 commit into
dennis-upbound merged 1 commit into
Conversation
The built-in SGLang mapping renamed running and waiting requests, KV usage and the token histograms, but none of SGLang's prefix cache or retraction counters. modelplane_prefix_cache_hits_total, modelplane_prefix_cache_lookups_total and modelplane_requests_preempted_total were empty for every SGLang deployment, so a prefix cache hit rate across vLLM and SGLang showed SGLang at nothing. SGLang v0.5.10.post1 and v0.5.12.post1, the versions the repo pins, publish cached_tokens_total, prompt_tokens_total and num_retracted_requests_total, which count what vLLM's prefix cache hits, lookups and preemptions count, in the same units. Counters have no buckets, so they share vLLM's names. SGLang counts when a request finishes where vLLM counts when it's scheduled, and labels its hits with a cache_source. The telemetry docs say so. sglang:queue_time_seconds stays unmapped: its buckets differ from vLLM's and between the two pinned SGLang versions (modelplaneai#523). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>
|
Docs preview: https://modelplane-docs-pr-544.vercel.app (ready once the site's Content workflow finishes) |
haarchri
approved these changes
Oct 20, 2026
haarchri
left a comment
haarchri
left a comment
Collaborator
There was a problem hiding this comment.
thanks for this update
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of your changes
The built-in SGLang mapping renames running and waiting requests, KV usage and the two token histograms, but none of SGLang's prefix cache or retraction counters.
modelplane_prefix_cache_hits_total,modelplane_prefix_cache_lookups_totalandmodelplane_requests_preempted_totalare empty for every SGLang deployment, so a prefix cache hit rate across a vLLM and an SGLang deployment shows SGLang at nothing. The comment above_SGLANGandtest_sglang_mappingssaid SGLang publishes none of them, checked against v0.4.9.post2. Both versions the repo pins now do.This maps the three counters onto vLLM's names:
sglang:cached_tokens_totalmodelplane_prefix_cache_hits_totalsglang:prompt_tokens_totalmodelplane_prefix_cache_lookups_totalsglang:num_retracted_requests_totalmodelplane_requests_preempted_totalCounters have no buckets, so #523 doesn't apply, and each pair counts the same thing in the same unit. SGLang counts when a request finishes where vLLM counts when it's scheduled, and
cached_tokens_totalhas acache_sourcelabel thatsum by (deployment)folds away. The telemetry docs now say both.sglang:queue_time_secondsstays unmapped. Its buckets differ from vLLM's and between the two pinned SGLang versions, which is #523.The metric names come from SGLang's source at v0.5.10.post1 and v0.5.12.post1. This didn't run against a live SGLang engine.
Fixes #543
I have:
nix flake check(or./nix.sh flake check) and made sure it passes.git commit -s.🤖 Generated with Claude Code