Skip to content

[None][feat] Add ZMQ KV-cache event tee with gap replay - #8

Merged
mohammed-deepinfra merged 4 commits into
deep-main-v1.3.0rc16from
mohammed/wip-kv-cache
Jun 23, 2026
Merged

mohammed-deepinfra merged 4 commits into
deep-main-v1.3.0rc16from
mohammed/wip-kv-cache

Conversation

@mohammed-deepinfra

@mohammed-deepinfra mohammed-deepinfra commented Jun 15, 2026 •

Copy link
Copy Markdown

[None][feat] Add ZMQ KV-cache event tee with gap replay

Description

Adds an optional read-only ZMQ publisher that tees TensorRT-LLM's native KV-cache events (including token_ids, which the SSE /kv_cache_events path drops) to an external consumer such as a Dynamo RadixTree indexer, in the vLLM ZMQ wire format.

The tee is driven synchronously from the existing kv_event_processor drain loop, so it preserves the single-consumer invariant and never disturbs the production SSE path. The PUB socket is bounded (SNDHWM) with non-blocking sends a missing or slow subscriber drops events rather than backing up the engine.

Because PUB/SUB is lossy, this also adds gap replay, ported to be byte-compatible with vLLM's ZmqEventPublisher protocol so an existing Dynamo / standalone-indexer replay client works unchanged:

  • A ring buffer (deque, default 10k batches) of recently published (seq, payload) pairs.
  • A ROUTER socket on a dedicated daemon thread. A subscriber that detects a sequence gap sends the missing start sequence; the publisher streams back every buffered batch from that point, then an END_SEQ=-1 sentinel.

@mohammed-deepinfra mohammed-deepinfra changed the title Mohammed/zmq publishing + replay vllm style [None][feat] Add ZMQ KV-cache event tee with gap replay Jun 15, 2026
Comment thread tensorrt_llm/commands/serve.py Outdated
multimodal_server_config: Optional[MultimodalServerConfig] = None,
served_model_name: Optional[str] = None):
served_model_name: Optional[str] = None,
kv_events_zmq_endpoint: Optional[str] = None,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

follow vllm format. kv-event-config dict instead of multiple params here

@mohammed-deepinfra
mohammed-deepinfra merged commit 1f7cc11 into deep-main-v1.3.0rc16 Jun 23, 2026
7 of 8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants