Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ dist/
# run directories, plans and result tables are evidence, not source
runs/
*.xlsx

# history evaluation results
.history
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,25 @@ Names are per project so several can be installed in one environment: this proje
cd hust-eval-client-mmlu-pro
python3 -m venv .venv && . .venv/bin/activate
pip install -U pip # REQUIRED on the system Python of macOS (pip 21): `pip install -e` fails without it
pip install -e . # core is standard library only
pip install -e '.[prepare]' # only for `mmlu-pro prepare` and `mmlu-pro materialize` (pyarrow, transformers)
pip install -e . # client plus pyarrow, transformers and jinja2 (needed by prepare/materialize)
hust-eval-mmlu-pro --help # or: python -m hust_eval_mmlu_pro --help
```

Python 3.9+. `mmlu-pro bench` additionally needs the `vllm` command on the client machine (it runs
`vllm bench serve`, which is a client).

**Pin `transformers` to the version you froze with.** The frozen prompts are token IDs produced by the model's chat
**Record the `transformers` version you froze with.** The frozen prompts are token IDs produced by the model's chat
template through `transformers`; another version may render the template differently and give different IDs, i.e.
a different plan. The first plan here was frozen with `transformers==5.14.1` and `pyarrow==19.0.1`; after
`prepare`, compare the `cases.json` SHA-256 with the one you expect (for the same model, parquet and reference, the
first plan's was `f77a7e5e6fd96fbbf46be7ab11f360309554105e97921805f02d1c93fc05fcf3`).
a different plan. `pyproject.toml` only sets floors (`transformers>=5.14.1`, `pyarrow>=19.0.1`, `jinja2>=3.1.0`), so
the versions actually in use come from the environment, and `receipt` writes them into `receipt/client.json`
(`client_packages`) next to the Python version. The first plan here was frozen with `transformers==5.14.1` and
`pyarrow==19.0.1`; after `prepare`, compare the `cases.json` SHA-256 with the one you expect (for the same model,
parquet and reference, the first plan's was
`f77a7e5e6fd96fbbf46be7ab11f360309554105e97921805f02d1c93fc05fcf3`).

Installation was verified with `pip install -e .` and `pip install .` into a fresh venv on Python 3.9 after
upgrading pip. The `[prepare]` extra was not installed from scratch in a clean environment.
upgrading pip, at a time when the default dependencies were empty. The current default set (pyarrow, transformers,
jinja2) was installed into fresh venvs on Python 3.11.6 and 3.12.13; Python 3.9 was not re-checked afterwards.

## Inputs you must provide

Expand Down
9 changes: 5 additions & 4 deletions docs/QUICKSTART.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,14 @@ git clone https://github.com/vLLM-HUST/hust-eval-client-mmlu-pro.git
cd hust-eval-client-mmlu-pro
python3 -m venv .venv && . .venv/bin/activate
pip install -U pip # Mac 自带 Python 的 pip 太旧,不升级的话下一行会失败
pip install -e '.[prepare]'
pip install 'transformers==5.14.1' 'pyarrow==19.0.1'
pip install -e . # 会一并装上 pyarrow / transformers / jinja2(prepare 与 materialize 需要)
hust-eval-mmlu-pro --version
```

`transformers` 的版本要和冻结提示时一致:提示是分词器按 chat template 渲染出的 token ID,版本不同可能渲染不同,
得到的就是另一份计划。升级 pip 之后的安装(`-e` 和普通安装)已验证;`[prepare]` 额外依赖没有在全新环境里从头装过(未验证)。
`pyproject.toml` 里给的是**下限**(`transformers>=5.14.1`、`pyarrow>=19.0.1`、`jinja2>=3.1.0`),不是钉死的版本;而冻结提示
对实际版本敏感 —— 提示是分词器按 chat template 渲染出的 token ID,版本不同可能渲染不同,得到的就是另一份计划。所以真正
用的是哪一版要记下来:`receipt` 会把 `pyarrow`/`transformers`/`vllm` 的实际版本写进 `receipt/client.json` 的
`client_packages`。第一次冻结用的组合是 `transformers==5.14.1` + `pyarrow==19.0.1`,要对齐这个哈希就先装这两版。

## 2. 取参考代码

Expand Down
12 changes: 7 additions & 5 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,13 @@ version = "0.1.0"
description = "MMLU-Pro client for an already-running OpenAI-compatible vLLM server: send requests, score, hash and seal evidence. Never starts a server."
readme = "README.md"
requires-python = ">=3.9"
dependencies = [] # core is standard library only

[project.optional-dependencies]
# only needed by `prepare` and `mmlu-pro materialize` (they read parquet and tokenise)
prepare = ["pyarrow==19.0.1", "transformers"]
dependencies = [
# `mmlu-pro prepare` and `mmlu-pro materialize` read the pinned parquet and render the chat template.
# Floors, not pins: the frozen prompts depend on the actual versions, which `receipt` records.
"pyarrow>=19.0.1",
"transformers>=5.14.1",
"jinja2>=3.1.0", # transformers does not declare it, but apply_chat_template fails without it
]

[project.scripts]
hust-eval-mmlu-pro = "hust_eval_mmlu_pro.cli:main"
Expand Down
Loading