Skip to content

fix: return hybrid cache block counts on the runtime branch - #73

Closed
hellolazy1024 wants to merge 2 commits into
stateaxis/runtime-0.19-baseline-20260828from
codex/fix-hybrid-cache-count-20261101
Closed

hellolazy1024 wants to merge 2 commits into
stateaxis/runtime-0.19-baseline-20260828from
codex/fix-hybrid-cache-count-20261101

Conversation

@hellolazy1024

@hellolazy1024 hellolazy1024 commented Nov 1, 2026 •

Copy link
Copy Markdown

A real Qwen3.5-35B-A3B TP=2 Ascend startup reached serving, but its first diagnostic request crashed in KVCacheManager.cache_blocks: num_cached_blocks > 0 received None. On this runtime branch, the base coordinator returns an integer while HybridKVCacheCoordinator discards manager return values.

Return the maximum manager count, matching this branch's base coordinator contract, while preserving aligned token limits, EAGLE lookahead clamping, and retention arguments. This targets the deployed runtime branch at 8ac6adf, not main: current main has a different cache/metrics contract.

Validation on the Ascend container using its isolated Python environment:

  • .venv/bin/python test_hybrid_cache_block_count.py: original source 3 failures; patched source 3 passed.
  • Tests cover maximum rather than sum, alignment/retention forwarding, EAGLE clamping, and zero/empty cases.
  • Serving recheck: a non-streaming diagnostic returned HTTP 200 with the expected text; the cache-count crash is gone. A separate local Ascend adapter keyword-forwarding fix was also required, so this is not a claim that this PR alone fixes every compatibility issue.
  • One private SWE-bench-Pro diagnostic reached 54 completed model calls before its 1200-second wall budget interrupted call 55; the cache-count crash did not recur. The saved partial patch was applied by the official grader but failed test startup, so this is not a successful task result. Eager mode and isolated Ascend compatibility changes were used; no benchmark performance or quality claim is made.

Duplicate check: open PR searches for cache_blocks return, hybrid cached blocks, and prefix cache crash found no matching fix; the scheduling admission PR #8 is unrelated.

AI assistance was used to diagnose, implement, and test this change. This is a draft for human review; no human line-by-line review or full repository test run is claimed. Private task prompts, patches, credentials, and raw experiment data are not included.

@ShuhaoZhangTony

Copy link
Copy Markdown
Member

2026-11-08 maintainer disposition: closing this draft without merging. It intentionally targets the retired runtime snapshot 8ac6adf; current main (782c0a75) has replaced that override with the base coordinator contract plus get_num_cacheable_tokens, and cache_blocks is again a side-effect-only API returning None. Applying the old “return max manager count” contract to current main would reintroduce a branch-specific API rather than fix a current defect. The supplied tests also construct private state with object.__new__, SimpleNamespace, and mocks, which current tests/AGENTS.md forbids. The branch and PR body retain the useful Qwen3.5/Ascend incident evidence; if that exact frozen runtime must remain supported, carry the patch in its versioned runtime branch and publish the serving receipt there, not in current main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants