推理加速

[dox012]

我主要使用本地部署的模型和一些线上模型的api进行推理
在测试时遇到了关于推理的一些问题：

1. 代码里好像没有进行并行加速，我试了一下单卡推理有点慢，想问下推理以及评测完整的测试集需要多少时间，我有8张显卡，如何进行推理加速？
2. 如果我只想推理和评测1000条中的前100条进行简单的实验该如何操作？
3. 我注意到生成回复时使用的是openai接口，而自动评估时使用的是vllm.llm，这两种方式有何区别？为何这两个要使用不同的接口呢？

[A-Quarter-Mile]

Thanks for your questions! You can flexibly adjust the inference setup based on your computational resources without affecting results.

If you only want to test a subset, you can directly filter the query file, but for a small-scale experiment we don’t recommend taking the first 100, please sample randomly instead.

In the repo, we provide two evaluation approaches: using a trained critic model or external SOTA models as judges. To stay consistent with our latest leaderboard, we recommend using the same judge setup as the current leaderboard (currently Claude-4.5).