Skip to content

Clarify vLLM P/D cached-token usage accounting - #41

Merged
sernote merged 2 commits into
mainfrom
codex/cache-watch-vllm-pd-accounting
Oct 8, 2026
Merged

sernote merged 2 commits into
mainfrom
codex/cache-watch-vllm-pd-accounting

Conversation

@sernote

@sernote sernote commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

OpenAI-shaped vLLM P/D usage can pass the inclusive-denominator check while D's cached-token count reflects KV transferred from P. The skill now checks the numerator's provenance and the usage-details emission flag before calling that count P-side prefix reuse. A proven remote override can still carry a real high P-side count.

The change adds a focused tag comparison and proxy contract in references/vllm.md, a short observability caveat, and durable behavioral cases 53/54. Scripts, normalized schemas and generic inclusive accounting are unchanged. The required deferred-token ceiling is remeasured in its existing test.

Evidence and relevance gate

vLLM v0.31.0 release lists PR #54222. Tagged source shows the P-side metadata write, the D-side truthy/int gate, and the remote-count override. The v0.30.0 scheduler and output processor lack that field. The proxy must carry P's kv_transfer_params to D with an integer remote_prefill_cached_tokens; missing or string values leave D's local count.

The CLI default is False for --enable-prompt-tokens-details, and the chat serving gate can omit details. Absence is unobserved, not zero. Source checked on 2026-10-07 at ced6857a / db9527a4.

Before this change, main's observability example mapped cached tokens generically, while its vLLM reference had no P/D usage provenance or emission caveat. This is an accounting guidance update, with no pricing or production changes.

Fresh verification output

python3 -m unittest discover -s tests -p 'test_*.py': 260 tests, OK
validate_skill_package.py audit-prompt-caching: status ok, no errors/warnings
run_trigger_eval.py audit-prompt-caching: status ok, 40 cases (30 positive / 10 negative)
Python compile over all bundled scripts and tests: 12 files OK
git diff --check: clean
bytecode cleanup and remaining-file check: no __pycache__ or .pyc

GitHub CI validation on published head 2193834 passed. The local verification output above was refreshed after resolving the main-branch conflicts.

The existing deferred-budget test first failed after the added guidance. The full corpus was remeasured after integration with main bb76302, its ceiling updated, then passed. Main → combined: 370,932 → 381,258 chars; aggregate ceil(chars/4) 92,733 → 95,315 tokens (+2,582). The separate per-file rounded sum is 92,747 → 95,329. The content delta remains exactly +10,326 chars; the earlier old-base estimate of +2,581 differs only because of rounding. No other test logic or budget changed.

Static evaluation

plugin-eval analyze fresh-main/combined and plugin-eval compare (0.1.2): score 59/D → 59/D, no new failures. Trigger estimate 152 → 152, invoked 6,700 → 6,700, deferred per-file estimate 92,747 → 95,329 (+2,582). Existing budget/structure findings remain; these are static estimates.

Behavioral evaluation and independent review

Separate blind before/after Porch sessions used frozen packages, identical prompts, no answer key, and claude-b / claude-opus-5-5 / high. Durable prompts 53/54 and the remeasured budget test are included in this patch.

Scenario Before Final after
53: v0.30.0 D near-total cached count versus P zero hits Rejects savings, but transfer mechanism remains a hypothesis Identifies D transfer accounting and denominator/provenance distinction; passive validation only
54: v0.31.0 proxy drops/retypes metadata and omits usage details Leaves exact field/type/fallback and flag default unknown Identifies integer round trip, local fallback and opt-in emission; missing is unknown
After-only positive control: proven high P count Not run Accepts the request's P-side count; separates flag/counter types and makes no performance, ROI or isolation claim

A separate read-only Porch review verified the source captures and contract, confirmed two wording findings, and assessed the targeted rubric. Both findings were fixed through Porch in one correction loop. Final focused independent review found no actionable findings, confirmed both corrections, independently reproduced the corpus counts, and verified all targeted scenario criteria (4/4, 5/5 and positive control 3/3). These single-sample audit answers and tagged-source checks do not demonstrate deployed behavior, measured savings/latency, rollout readiness or complete tenant/application isolation. Backports, forks, other releases and non-chat endpoints require their own evidence.

Execution plans, primary-source captures, prompts, full outputs and deviation journals are preserved locally outside the checkout and are excluded from this PR.

Main integration on 2026-10-08

Merged pinned main bb76302 into the same feature branch with merge commit 2193834, preserving history. Conflicts were limited to the eval append and deferred-budget commentary/constant. All 54 cases are preserved exactly: main's 51/52 and this PR's 53/54, with shared cases unchanged. Gemini guidance matches main byte-for-byte; vLLM and observability guidance match the accepted PR byte-for-byte. Test logic is unchanged apart from the measured constant.

A separate read-only claude-b Opus 5.5 high Porch review independently verified the union, file scope, parent snapshots and combined corpus measurements. No repository findings remained. All AGENTS checks were rerun successfully on the combined tree. The earlier behavioral results above remain historical scenario observations; this conflict repair introduces no behavior change and does not manufacture production evidence.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 7, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T09:21:05.649203Z 8eda94f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@sernote
sernote merged commit 5ab93a3 into main Oct 8, 2026
1 check passed
@sernote
sernote deleted the codex/cache-watch-vllm-pd-accounting branch October 8, 2026 00:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant