Model → Graph → Operator → Engine → Backend → Device
模型 → 执行图 → 算子 → 引擎 → 后端 → 设备
RouteInfra is an open-source, self-hosted control plane that chooses an executable LLM inference plan from live device state, loaded models, request SLOs, KV state, and reproducible runtime evidence.
RouteInfra 是一个开源、自托管的常驻式 LLM 执行控制平面：根据实时设备状态、已加载模型、请求 SLO、KV 状态和可复现实机证据，自动选择完整推理执行计划。
Warning
RouteInfra is an active 0.1 foundation release. Released-checkpoint parity, real external-engine matrices, PostgreSQL/Redis recovery, and GPU Kubernetes validation are still beta gates.
RouteInfra 当前是活跃开发中的 0.1 基础版本。真实 checkpoint 对齐、外部引擎矩阵、PostgreSQL/Redis 恢复和 GPU Kubernetes 验证仍属于 beta gate。详见实施状态。
Most inference routers choose a model. Most serving platforms choose a runtime. RouteInfra chooses the whole execution plan and refuses to promote an unverified path into production.
多数推理路由器只选择模型，多数 serving 平台只选择运行时。RouteInfra 选择的是完整执行计划，并且不会把未经真实验证的路径静默升级为生产能力。
request
├─ hard gates: artifact / quality / memory / SLO / lease / receipt
├─ Pareto candidates: model + RouteIR + operator + engine + backend + node
├─ deterministic or offline-learned ranking
└─ signed short-lived plan → direct Agent execution → Route Receipt
| RouteInfra | What it means / 含义 |
|---|---|
| Always-on control plane | Observes nodes, loaded models, queues, memory pressure and KV state. / 持续观察节点、模型、队列、显存压力和 KV 状态。 |
| Evidence gates | Importability is not capability; load and generation must succeed on the target device. / 能 import 不等于可用，必须在目标设备真实加载并生成。 |
| RouteIR | Full-graph IR for layout, memory, kernels and placement. / 面向完整执行图、布局、内存、算子和放置的中间表示。 |
| Direct execution | Controller signs a short-lived plan; the SDK calls the selected Agent. / Controller 签发短期计划，SDK 直连选中的 Agent。 |
| Auditable failure | No feasible candidate returns structured NO_FEASIBLE_PLAN. / 无候选时结构化失败，不静默降级。 |
| Research continuity | FusionInfer KV, scheduler, benchmark and evidence paths remain first-class. / 原 FusionInfer 的 KV、调度、benchmark 和 evidence 方向继续保留。 |
RouteInfra is inspired by RouteLLM's candidate generation, quality prediction and threshold calibration. It extends the routed object from “which model?” to “which verified infrastructure execution plan?”.
RouteInfra is intentionally distinct from runInfra: its primary object is an open-source, self-hosted, continuously running node-and-request control plane with live state, signed plans, fail-closed receipts, RouteIR and cross-layer KV semantics—not merely a benchmark or deployment workflow.
RouteInfra discovers installed components through manifests and Python entry points, then compiles only protocol-compatible compositions for the current device. The first integration boundary is deliberately truthful:
| Layer / 层 | Components / 组件 |
|---|---|
| Engine + scheduler | FusionInfer, vLLM, SGLang |
| KV store | FusionInfer KV, LMCache, FlexKV, Mooncake Store |
| KV transport | Mooncake Transfer Engine, RDMA/TCP/NIXL where verified |
| Kernel / compiler | Torch, Inductor, Triton, CUDA/HIP native kernels |
| Device backend | CUDA, ROCm, Ascend, CPU |
Importing a package is only discovery. A composition becomes verified only
after real model load/generation and an independent receipt bound to model,
layout, runtime, driver, topology, and device. See the plugin protocol
and compatibility matrix.
flowchart LR
C[Client / SDK<br/>客户端] --> CTRL[Controller<br/>全局控制平面]
CTRL --> R[Evidence Router<br/>证据路由器]
R --> P[Signed Plan<br/>签名执行计划]
P --> A[Node Agent<br/>节点代理]
A --> E[Engine Adapters<br/>引擎适配器]
E --> X[RouteIR / FusionInfer / vLLM / SGLang / Ollama]
A --> K[KV Service<br/>KV 服务]
CTRL --> S[(SQLite / PostgreSQL)]
CTRL -. short state .-> Q[(Redis)]
A --> D[GPU / CPU / Node<br/>设备与节点]
logical model group
→ model revision / quantization variant
→ RouteIR execution profile
→ operator and kernel set
→ inference engine
→ backend API
→ device / node
python -m venv .venv
source .venv/bin/activate
pip install -e '.[dev,serve]'
pytest -q
routeinfra-doctor --model .ci_tiny_modelOptional control-plane and cluster dependencies / 可选控制平面与集群依赖：
pip install -e '.[controller,postgres,kubernetes]'doctor 只证明运行时命令可执行；生产 capability 仍必须通过真实模型加载和生成验证。
export ROUTEINFRA_PLAN_SECRET="replace-with-at-least-32-random-bytes"
routeinfra-controller --database ./routeinfra-controller.db --host 127.0.0.1 --port 8080In another terminal / 在另一个终端：
export ROUTEINFRA_PLAN_SECRET="replace-with-at-least-32-random-bytes"
routeinfra-agent --node-id workstation-1 --capacity-bytes 16000000000 --host 127.0.0.1 --port 8081The SDK asks the Controller for a signed plan and sends the prompt only to the selected Agent:
from routeinfra import RouteInfraClient, RouteRequest, SLOConstraints
client = RouteInfraClient("http://127.0.0.1:8080")
response = client.chat(RouteRequest(
request_id="request-1",
model_group="code-chat",
prompt="Write a bounded queue in Python.",
input_tokens=8,
output_tokens=128,
slo=SLOConstraints(max_ttft_ms=500, min_quality=0.85),
))By default, the Controller stores a prompt digest and routing features—not the raw prompt. 默认情况下，Controller 只保存 prompt 摘要和路由特征，不持久化原始 prompt。
RouteIR imports complete PyTorch FX or torch.export graphs. Nodes without native kernels remain represented and lower to Torch/Inductor rather than disappearing into an invisible fallback.
RouteIR 导入完整的 PyTorch FX 或 torch.export 图。没有原生 kernel 的节点仍保留在 RouteIR 中，并 lowering 到 Torch/Inductor，不会消失在不可审计的 fallback 中。
from routeinfra.ir import RouteCompiler, RouteIRExecutor, check_routeir_parity
artifact = RouteCompiler(".routeinfra-cache").compile(
model, example_inputs, architecture="qwen2", backend="torch"
)
output = RouteIRExecutor(artifact).run(*example_inputs)
report = check_routeir_parity(artifact, example_inputs, reference=model)
assert report.passedInitial architecture contracts are llama, qwen2 and mixtral. Checkpoint-level parity is a release gate, not inferred from an architecture string.
首批架构契约包括 llama、qwen2 和 mixtral。真实 checkpoint 对齐属于 release gate，不能仅凭架构字符串推断。
- Model artifact, tokenizer, runtime, device and execution-profile identity.
- Receipt validity and live node heartbeat.
- Quality evidence for cross-model or non-equivalent routes.
- Memory, cold-start, TTFT, TPOT, latency and cost constraints.
- Pareto frontier construction, then deterministic/offline ranking.
- Shadow-only bandit observation.
如果没有候选满足约束，API 返回带候选拒绝原因的结构化 no_feasible_plan，不会静默降级。
Production KV reuse requires an exact fingerprint over model revision, tokenizer, layer/head geometry, dtype, layout, block size and positional encoding. Same-engine reuse is the verified path; cross-engine conversion is explicitly experimental.
生产 KV 复用要求对模型 revision、tokenizer、层/头结构、dtype、layout、block size 和位置编码做精确指纹匹配。同引擎复用是 verified 路径；跨引擎转换明确属于 experimental。
The original FusionInfer direction remains under routeinfra.fusion_infer: continuous batching, paged/radix/hybrid KV, prefill/decode scheduling, request telemetry and benchmark schema v2.
原有 FusionInfer 方向继续位于 routeinfra.fusion_infer，包括 continuous batching、paged/radix/hybrid KV、prefill/decode 调度、请求级 telemetry 和 benchmark schema v2。
| Surface / 模块 | Status / 状态 |
|---|---|
| Contracts, receipts, SQLite, Evidence Router | Verified by tests / 测试验证 |
| FusionInfer KV/scheduler/benchmark | Preserved and tested / 保留并通过测试 |
| RouteIR generic FX/export execution | Verified on synthetic graphs / synthetic 图已验证 |
| Ollama, vLLM, SGLang adapters | Lifecycle implemented; exact model receipts pending / 生命周期已实现，精确模型 receipt 待补 |
| TensorRT-LLM | Current host unavailable due ABI mismatch / 当前设备因 ABI 不兼容不可用 |
| PostgreSQL, Redis, Kubernetes | Surfaces implemented; live validation pending / 接口已实现，实机验证待补 |
The local suite count is reported in Implementation Status; it is a regression signal, not a production-readiness claim. See Implementation Status, Architecture, Roadmap, and Evidence Contract.
routeinfra/
adapters/ runtime probes and managed engine processes
agent/ node lifecycle, leases and dispatch
api/ Controller and Agent HTTP APIs
control/ signed plans and control-plane service
ir/ RouteIR, passes, lowerings, parity and autotuning
kv/ layout negotiation and interchange
routing/ evidence router and learning policies
fusion_infer/ preserved KV/scheduler research runtime
benchmark/ reproducible evidence pipeline
deploy/ CRDs and Helm chart
tests/ contract, runtime and integration tests
docs/ architecture, evidence and migration records
Please read CONTRIBUTING.md before opening an issue or pull request. Do not label a proxy, mock, import check, tiny fixture, or CPU smoke as production performance evidence. Every promoted capability needs reproducible receipts.
提交 issue 或 pull request 前请阅读 CONTRIBUTING.md。proxy、mock、import 检查、tiny fixture 或 CPU smoke 都不能被标记为生产性能证据。任何晋升为 verified 的能力都必须提供可复现 receipt。
RouteInfra is released under the Apache License 2.0. / RouteInfra 使用 Apache License 2.0 开源。