A benchmark measuring how output length prediction quality and calibration affect KV-cache-aware scheduling in LLM serving.
For full methodology and design decisions see design.md.
- How much does output length prediction improve scheduling?
- Is a noisy but calibrated predictor as good as a perfect one?
7 predictors × 4 safety margins × 3 scheduling policies × 4 arrival rates
Predictors: oracle, noisy_10, noisy_30, noisy_50, mean_only, prompt_ratio, none
Safety margins: 1.0, 1.1, 1.25, 1.5
Scheduling: greedy, memory_first, predictive_admission
Best effective throughput per predictor under memory_first:
Predictor best_margin throughput vs greedy oracle 1.5 5.78 req/s +44% noisy_10 1.5 5.76 req/s +44% noisy_30 1.5 5.65 req/s +41% noisy_50 1.5 5.40 req/s +35% prompt_ratio 1.5 4.52 req/s +13% mean_only 1.0 4.14 req/s +3% none 1.0 0.00 req/s catastrophic
Greedy baseline: ~4.01 req/s regardless of predictor.
Calibrated prediction matters more than raw predictor accuracy. noisy_10 @ 1.5x margin is nearly identical to oracle @ 1.5x margin. 5.76 vs 5.78 req/s effective throughput. 99.7% of oracle gain.
The optimal safety margin is 1.5x across almost all predictors. Underestimating output length causes KV overflow and preemption. Overestimating causes rejection. The asymmetry favors overestimation.
Greedy is insensitive to prediction quality. All greedy configurations produce 4.0-4.1 req/s regardless of predictor. Prediction only helps when paired with admission-control-based scheduling.
The biggest gain is from having any calibrated prediction at all. None to mean_only: +500x effective throughput. Mean_only to noisy_30: +36%. Noisy_30 to oracle: +2.3%.
predictive_admission is a practical middle ground. Matches memory_first with calibrated predictions. When predictions are wrong, preempts gracefully instead of rejecting.
Raw prediction quality and scheduling calibration are separate concerns. A noisy predictor with a good safety margin outperforms a precise predictor without calibration in real scheduling scenarios.
output-length-predictor-bench/ ├── src/ │ ├── init.py │ ├── config.py │ ├── workload.py │ ├── predictor.py │ ├── simulator.py │ └── analysis.py ├── results/ ├── plots/ ├── LICENSE ├── design.md ├── README.md ├── requirements.txt └── run.py
python3 -m venv venv source venv/bin/activate pip install -r requirements.txt python run.py
Outputs: results/results.csv results/summary.txt plots/effective_throughput.png plots/margin_sweep.png plots/preemption_vs_predictor.png
This project closes a three-part loop:
sharegpt-workload-bench: real output length distributions are heavy-tailed kv-cache-aware-scheduler: scheduling using predicted KV footprint output-length-predictor-bench: where does the prediction come from, and does it matter?
MIT License. See LICENSE for details.
Joao Felipe De Souza 2026