Repository navigation
[Observability] Add minimal OffloadingConnector transfer/recovery observer seams #43
Description
背景与范围
落实 vLLM-HUST/vllm-hust-kv-transfer-observability#2 的真实宿主接入要求：
vLLM-HUST/vllm-hust-kv-transfer-observability#2 (comment)
插件 PR #5/#6 已合并，提供 typed observation、bounded sink、descriptor capture、normalizer、显式 binding/config 和 CPU/fixture 测试；尚无真实宿主 KV 事件绑定，保持 import_only。本 Issue 只请求最小、通用、默认关闭的宿主观察出口；exporter、队列、规范化和文件发布继续在插件仓库，不迁回 core。
现有方案：
https://github.com/vLLM-HUST/vllm-hust-kv-transfer-observability/blob/53e69013dcf27fd08a93d142288c93c701edabd3/docs/current_host_seam_proposal.md
该链接为已合并旧版；以下内容是 2026-09-28 针对新代码的修正，插件本地提案也已更新但尚未推送。
已核查，不是重复 EventBus
- core main：c7b61770097b5c6e9ae11d96baef59e3689231f1
- Ascend main：17f68177472b896facf3f0501d424d24e16e699a
- Manager main：701aa95ae8d5b23b5ea8c8ec475ceaa470393e54
core #3 与 #6 尚未合并；#6 head 1287794 提供 request finished/preempted/reclaimed，不提供 KV transfer/recovery/H2D/first-compute。#3 是 sampled-output 生命周期。应复用它们最终合一的 registry，不能新增第三套总线。当前 general-plugin loader/evidence 证明加载/调用，不证明 KV transfer。BlockStored/BlockRemoved 是 residency 事件，不能直接改名替代。
已检索本仓 KV observer/offloading 与 upstream offloading observer；未发现实现本完整接口的重复工作。关联本仓 #1 的最小宿主 seam 方向。
首个实际目标
OffloadingConnector 的 CPU↔设备 offload。Ascend 已注册 AscendOffloadingConnector，并将 CPUOffloadingSpec/TieringOffloadingSpec 映射到 NPU 实现；复用现有 scheduler/worker 逻辑，非整体移植 legacy 补丁。首个验证配置的 runner、TP、模型和软件版本需要在实际运行前明确，不声称全面兼容。
最小接口与调用点
- 共享注册/注销：在已有共享 outlet 上注册观察 sink，返回可移除关联；禁用无事件构造/计时/I/O，观察异常不能改变 serving。宿主仅发布闭合 primitive fields，不 import 插件类型。
- scheduler→worker 身份：复用 offloading/common.py 的 TransferJob / OffloadingConnectorMetadata，携带 bounded request/job/recovery-episode 上下文；worker 明确 rank/generation。不通过时间接近或文件名猜身份。
- 实际提交：offloading/worker.py 的 start_kv_transfers / handle_preemptions 中真实 submit_load/submit_store 的结果。prepare_store_kv 只是排队，不能作为传输 start。replicated-layout 非写入 worker 的 mark_completed 是确认，不是物理 D2H。
- 完成与回执：在 get_finished 的实际 TransferResult 处观察，利用 OffloadingWorkerMetadata 回传逐 worker 的 bounded H2D receipt。现有 completed_jobs 仅聚合 job→count，不足以证明 rank/generation 覆盖；观察字段不能改变原有调度计数语义。
- recovery/admission：scheduler update_connector_output 聚合回执，实际恢复调度时发布 admission，并向 worker 传递精确成功 H2D roster。普通 prefix-cache load 不是自动等于 post-preemption recovery；完成/具备调度资格也不等于实际 admission。
- first compute：V1 对应 core/Ascend 的真实 _model_forward；V2 对应 core gpu/model_runner.py eager/piecewise/fullgraph 执行分支，Ascend V2 委托该路径。ActiveKVConnector.pre_forward 也会被 no_forward 调用，不能仅在此无条件发 first-compute；排除 dummy/profile/capture、padding、重复或过期 generation。异步 load 可能在当前批 forward 之后提交，须关联以后真正执行该恢复请求的批次。
- failure/cancellation：当前 offloading worker 会 assert 提交/完成成功；在实际失败/放弃处观察但不吞掉原本的宿主错误，也不新增恢复策略。request_finished 后仍可能有合法存储任务，不可一律标 cancellation。进程消失但未发回执只能算不完整证据。
- descriptor 安全边界：发布 region ID、相对 offset/size/direction 等有界布局，不暴露原始 tensor、process/device address、token/hash 或 KV 内容。Ascend native/cpu_npu.py 的 batch_src/batch_dst 是原始指针数组，不能传给插件再过滤。文件 I/O 在插件有界后台 writer 中执行。
验证与外部依赖
- disabled 等价性，observer 异常隔离，真实提交/完成顺序；
- 跨进程/rank/generation、重复/丢失/乱序与精确 receipt roster；
- 正常/失败/实际取消，注销及退出清理；
- exact host/plugin 版本上的 Manager discover/check/plan/run 和目标 worker 覆盖；
- 真实设备回执及启停正确性/开销，不以 fixture 代替设备证据，不要求加速曲线。
当前两边最新 main 的 RoutedExpertsLists/RoutedExpertsTensors 导入不匹配仍存在，关联 vLLM-HUST/vllm-ascend-hust#34。Ascend 仓库记录的 upstream verified core ced6857 是待验证候选，不能把它直接称为已通过运行测试的组织 core 版本。
这是一项具体实现跟踪，不申请额外联签或冻结记录。请沿 #3/#6 的共享 outlet 协调宿主最小调用点；插件实现及诊断工具工作独立推进。
AI assistance used for code inspection and drafting; no host source changes or hardware validation are claimed in this issue.
Activity
xiehanlong834-gif commentedon Sep 29, 2026
宿主侧最小出口已实现并提 PR：#46（base
main，head1d55929），按本 issue 请求的形态：不新增总线、宿主不 import 插件类型、默认关闭、fail-open。已覆盖：
register_kv_transfer_observer返回可移除 handle，unregister幂等；无 observer 时单次 dict 守卫（不读时钟、不建记录、不回调）。start_kv_transfers/handle_preemptions里真实submit_load/submit_store成功后发布；prepare_store_kv只排队不发。get_finished的真实TransferResult（success/bytes/duration）处发布。host_shutdown）。mark_completed是确认不是物理 D2H，不发布。尚未包含（我这边下一步）：descriptor v2 区域清单、recovery epoch/requeue/admission 与逐 worker H2D receipt roster、first-compute（core+Ascend）。
验证：
e521b42e+ 该分支，tests/v1/kv_connector/unit/offloading_connector/472 passed / 2 skipped（CPU 测试模式），ruff clean。feat(kv-offload): report H2D receipts and recovery requeue/admission
xiehanlong834-gif commentedon Sep 29, 2026
第二块也已提交（同一 PR #46，新 commit
e89a7e5）：OffloadingWorkerMetadata.load_receipts回传逐 worker 的 bounded H2D receipt（job → sorted ranks），调度侧在 pending 归零时发布 receipt（带精确 rank 集合）。原有completed_jobs计数语义未改。preempted_req_ids）打开一个 recovery episode 并发 requeue，epoch 取request.num_preemptions；调度器内部抢占原因对 connector 不可见，reason 保持unclassified（不猜）。episode 内完成的 H2D load 收集为 roster；只有调度器真正 resume（resumed_req_ids）且 roster 精确时才发 admission。普通 prefix-cache load 不会变成 recovery。prepare_store_kv的排队都不算传输点。验证：
e521b42e+ 该分支，tests/v1/kv_connector/unit/offloading_connector/485 passed / 2 skipped（新增 13 个用例），ruff clean。仍缺：descriptor v2 区域清单、first-compute（core+Ascend）。
feat(kv-offload): publish bounded region-relative transfer layouts
xiehanlong834-gif commentedon Sep 29, 2026
第三块（descriptor v2 区域布局）也已进 PR #46（新 commit
537b7f0）：v1/kv_offload/cpu/gpu_worker.py的 op 构建处）直接产出 region-relative 描述符：region id = canonical tensor 索引（对应 legacy 的src_tensor_N/dst_tensor_N），offset = 指针减各自 region base，size = 每 op 字节数。绝对指针、tensor 对象、hash、token id 一概不出边界——不需要插件再做二次过滤（对应本 issue 第 8 条：Ascendbatch_src/batch_dst不能原样外传）。layout为 None）。cpu_npu.py自建 op）是 follow-up——同一 API，调用点要在 vllm-ascend-hust 里补。验证（
e521b42e+ 该分支，CPU 测试模式）：offloading_connector/+test_region_descriptors.py490 passed / 2 skipped（新增 18 个用例）；test_gpu_worker.py需真卡，在与不含本改动的main上同为 32 failed（既有设备依赖失败，非回归）；ruff clean。宿主侧至此只剩 first-compute 一个缺口。
xiehanlong834-gif commentedon Sep 29, 2026
Host-side update for #43: the last open item (first compute) is up on the
feat/kv-offload-observability-seambranch / PR #46.first_computeis published when an admitted request actually reaches a real model forward, and consumed exactly once. Dummy, profile, and graph-capture batches never report._model_forward, the point dummy runs never reach; on V2 it is guarded bynot dummy_runin both the FULL and the piecewise/eager branches.With this, all eight host-side items of #43 have a call site in core except two Ascend-side follow-ups, both of which need edits in
vllm-ascend-hust(the descriptor call site incpu_npu.py, and the one-line observation in the Ascend V1execute_model, which overrides the core runner). The Ascend V2 path delegates to core and is already covered.xiehanlong834-gif commentedon Sep 29, 2026
Host side is complete and open for review: #46 (4 commits,
1d55929→98e437fd; head CI green, checks taken onorigin/main@e521b42e+ the four commits)Item-by-item against the eight points above:
offloading/observability.py:register_kv_transfer_observerreturns a removable handle,unregisteris idempotent, dispatch is fail-open (a raising observer is logged, removed, and serving continues). Disabled costs one dict guard per call site: no clock read, no record built, no callback. The host imports no plugin type, and this adds no bus of its own - no thread, no queue, no file I/O in core. It is a connector-local registry; when the registry from feat(observability): expose default-off request lifecycle events #6 lands this surface can re-home onto it with the emit points unchanged. This is the one design call I would like your read on.job_id+request_idfromTransferJobplus an explicitrank; recovery context travels inside the existingOffloadingConnectorMetadataasrecovery_admissions = {req_id: (epoch, roster, kind)}(bounded 4096). Nothing is inferred from time proximity or filenames;epochis the request's ownnum_preemptions. Gap I will not paper over: the host has no worker generation identifier today. If you want one, it is a small bounded counter and I would rather add it explicitly than leave it implied.worker.py::start_kv_transfersandhandle_preemptions(realsubmit_load/submit_store).prepare_store_kvonly queues, and a replicated-layout non-writermark_completedis a confirmation, so neither is observed.TransferResultinworker.py::get_finished; per-worker H2D receipts travel through a newOffloadingWorkerMetadata.load_receipts = {job_id: ranks}and aggregate inscheduler.py::update_connector_output, publishing once every owing worker reported. Existingcompleted_jobscounters are untouched; the recovery epoch currently rides on the requeue/admission records rather than the receipt (say the word if you want it on both)._observe_recovery_requeuedfires onpreempted_req_ids,_observe_recovery_admittedonly on a realresumed_req_idsresume carrying the exact sorted roster. A plain prefix-cache load never reports an admission, and completing a load is never reported as a resume. The exact roster goes to the workers in the same step's metadata. Requeue reason isunclassified: the connector cannot see the scheduler-internal reason, and I preferred that over a guess._model_forwardingpu_model_runner.py, a point dummy, profile, and capture runs never reach; V2:gpu/model_runner.pyafterpre_forwardin both the FULL and piecewise/eager branches, guarded bynot dummy_run.pre_forwarditself is never used for this, per yourno_forwardnote. Exactly once per episode: the worker stashes the admission when the metadata is bound and pops it on the first real forward; a newer admission replaces a pending one, and finished requests drop theirs. The async case holds because the marker resolves on the batch that actually forwards the request, not on the batch that submitted its load. The prefill/decode kind comes from the request's computed-token state at admission, not from a token-count heuristic that speculative decoding would distort.shutdown(host_shutdown), and nothing is marked cancelled merely because a request finished. On failures I have to be plain: the offloading workerasserts success in three places, so a failed submit or result aborts before any observation point exists. Adding one would change host error semantics, which this item explicitly does not ask for.v1/kv_offload/cpu/gpu_worker.py::build_region_descriptors: region id is the canonical tensor index (the numeric form of the legacysrc_tensor_N), offsets are relative to the region base, the inventory is bounded at 4096 withdropped_descriptorscounted. No raw tensors, process/device addresses, token ids, or hashes leave the copy site, and all file I/O stays in the plugin.Two Ascend follow-ups, both edits in
vllm-ascend-hust, not here:cpu_npu.py- the Ascend handler builds its own copy ops;NPUModelRunneroverrides bothexecute_modeland_model_forward, so the core hook does not run there. The Ascend V2 path delegates to core and is already covered.What this does not claim: no device evidence, no Manager discover/check/plan/run, no overhead numbers. Validation so far is CPU component tests (500 passed / 2 skipped on a 910B2 node in CPU mode,
VLLM_TARGET_DEVICE=cpu) plus green CI. The plugin should stayimport_onlyuntil a real event chain runs.Ask: (a) is the item-1 shape acceptable - connector-local registry now, re-homed onto the #6 registry when it merges; (b) do you want the recovery epoch on receipt records, and a worker-generation field? Both are small follow-up commits if yes.
ShuhaoZhangTony commentedon Nov 8, 2026
2026-11-08 project-owner acceptance: closing the host-observability seam as completed. The later accepted integration supersedes the original PR #46 path: Host PR #53 merged the versioned observer contract and Plugin PR vLLM-HUST/vllm-hust-kv-transfer-observability#15 merged the descriptor consumer, with fail-closed version admission, identity/replay checks, packaging, and 259 local tests plus remote CI. Current main also contains dedicated offloading observability and recovery tests. Real hardware/performance qualification remains correctly tracked in the plugin's issue #2; it is not a reason to keep this host-interface request open.