inference-engine-ops

A Rust-native LLM inference server built from scratch — continuous batching, KV cache management, semantic caching, and quantization benchmarking, with every optimization measured and documented. Not a wrapper around vLLM; an implementation of what vLLM does internally.

Status: 🚧 In development — M3 (epoch-level continuous batching)

Why this exists

Most portfolio LLM projects call an API and call it a day. This one answers a narrower, harder question: what actually happens between "a request comes in" and "tokens stream out," and how do you make that fast and cheap under concurrent load — without touching the model itself.

Headline results

(filled in as milestones complete — see docs/BENCHMARKS.md)

| Metric | Baseline | Optimized | Source | |---|---|---|---|---| | Throughput (tok/s, single) | 1.17 | 9.30 (7.9x) | docs/BENCHMARKS.md | | Throughput (tok/s, B=4) | — | 10.80 | docs/BENCHMARKS.md | | Throughput (tok/s, continuous) | — | 5.66 | docs/BENCHMARKS.md | | Latency (TTFT, single) | 0.20s | 0.25s | docs/BENCHMARKS.md | | Cache hit rate | — | TBD | docs/BENCHMARKS.md | | GPU cost / 1k tokens | TBD | TBD | docs/BENCHMARKS.md |

Architecture

CLI/Queue → Scheduler/Batcher (in-process)
                              ↓
                    KV cache ← → Model executor (Candle)
                              ↓
                    Token decoder → Text output

Axum HTTP layer, Redis semantic cache, Prometheus metrics — planned for M5+.

Full design rationale in docs/ARCHITECTURE.md.

Components

| Component | What it does | Status | Design doc | |---|---|---|---|---| | HTTP layer | OpenAI-compatible /v1/chat/completions, streaming via SSE | ❌ Not started | — | | Scheduler | Naive batching (fixed B, synchronized) | ✅ M2 | src/scheduler/mod.rs | | Scheduler | Continuous batching (queue, epochs) | ✅ M3 | src/scheduler/continuous.rs | | Scheduler | Continuous batching (per-step, interleaved prefill/decode) | ⏸ M4 | — | | KV cache manager | Naive (Candle built-in) | ✅ M1 | src/model/mod.rs | | KV cache manager | Paged block-based | ⏸ M4 | src/kv_cache/paged.rs | | Semantic cache | Redis-backed similarity match | ❌ Not started | — | | Quantization bench | FP16 vs GGUF Q4/Q8 comparison | ❌ Not started | docs/BENCHMARKS.md | | Observability | Prometheus + Grafana | ❌ Not started | — | | Load testing | k6/vegeta concurrent load | ❌ Not started | load_test/ |

Quick start

# Clone and build
git clone https://github.com/0xfave/inference-engine-ops.git
cd inference-engine-ops
cargo build --release

# Single request
cargo run --release -- --prompt "what is rust?" --max-tokens 100

# Batch request (4 identical prompts, synchronized)
cargo run --release -- --batch-size 4 --prompt "what is rust?" --max-tokens 100

# Continuous batching (queue, different prompts, epoch-level scheduling)
cargo run --release -- \
  --mode continuous \
  --prompts "what is rust?,hello world,capital of france?,tell me a joke,test" \
  --max-tokens 50 \
  --max-batch 4

Repo structure

inference-engine-ops/
├── src/
│   ├── api/              # Axum routes (not started)
│   ├── scheduler/
│   │   ├── mod.rs        # NaiveBatcher (M2) — fixed-batch batching
│   │   └── continuous.rs # ContinuousBatcher (M3) — epoch-level queue scheduler
│   ├── kv_cache/
│   │   ├── mod.rs
│   │   ├── naive.rs      # (empty — using Candle's built-in Cache for now)
│   │   └── paged.rs      # (empty — planned for M4)
│   ├── model/mod.rs      # ModelExecutor — Candle Llama wrapper
│   ├── cache/mod.rs      # Redis semantic cache (not started)
│   ├── metrics/mod.rs    # Prometheus instrumentation (not started)
│   └── lib.rs
├── docs/
│   ├── PRD.md            # Full product requirements, milestones M0–M8
│   └── BENCHMARKS.md     # Measured results + design decisions
├── README.md
└── Cargo.toml

Tech stack

Rust · Axum · Tokio · Candle · Redis · Prometheus · Grafana · Docker

Milestones

#	Milestone	Status	Result
M0	Baseline (no KV cache, single request)	✅ Done	1.17 tok/s
M1	Naive KV cache	✅ Done	9.30 tok/s (7.9x)
M2	Naive batching (fixed B=4)	✅ Done	10.80 tok/s (+55% over B=1)
M3	Continuous batching (epoch-level queue)	✅ Done	Queue + slot lifecycle demonstrated
M4	Paged KV cache + per-step continuous batching	⏸ Next	—
M5	Axum HTTP server, SSE streaming, OpenAI API	❌	—
M6	Semantic cache (Redis)	❌	—
M7	Prometheus metrics + Grafana	❌	—
M8	Quantization benchmarking (GGUF)	❌	—

Full details in docs/PRD.md and docs/BENCHMARKS.md.

Non-goals (v1)

Multi-model serving, SGLang/TensorRT-LLM integration, autoscaling, billing simulation, A/B testing, canary deploys, Kubernetes, agent orchestration. See docs/PRD.md for rationale.

Writeups

• Dev phase (architecture, batching/cache design decisions) — coming soon
• Prod phase (AWS deploy, cost analysis) — coming soon

License

MIT