Skip to content

Commit 181d71b

Browse files
raytrunraytrun
committed
Release data split manifests
1 parent 8ebede0 commit 181d71b

29 files changed

Lines changed: 13545 additions & 4 deletions

File tree

‎.gitignore‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,20 @@ build/
55
dist/
66
site/
77

8-
data/
8+
data/*
9+
!data/README.md
10+
!data/searchqa_id_split/
11+
!data/searchqa_id_split/**
12+
!data/livemathematicianbench_id_split/
13+
!data/livemathematicianbench_id_split/**
14+
!data/docvqa_id_split/
15+
!data/docvqa_id_split/**
16+
!data/officeqa_id_split/
17+
!data/officeqa_id_split/**
18+
!data/spreadsheetbench_id_split/
19+
!data/spreadsheetbench_id_split/**
20+
!data/alfworld_path_split/
21+
!data/alfworld_path_split/**
922
outputs/
1023
logs/
1124
external/

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ python scripts/train.py \
139139
# Train on ALFWorld:
140140
python scripts/train.py \
141141
--config configs/alfworld/default.yaml \
142-
--split_dir /path/to/your/alfworld_split \
142+
--split_dir data/alfworld_path_split \
143143
--azure_openai_endpoint https://your-resource.openai.azure.com/ \
144144
--optimizer_model gpt-5.5 \
145145
--target_model gpt-5.5

‎configs/alfworld/default.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ env:
1919
name: alfworld
2020
skill_init: skillopt/envs/alfworld/skills/initial.md
2121
split_mode: split_dir
22-
split_dir: data/ablation_splits/alfworld/2-1-7_seed42
22+
split_dir: data/alfworld_path_split
2323
data_path: ""
2424
split_output_dir: ""
2525
max_steps: 50

‎configs/livemathematicianbench/default.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ env:
99
name: livemathematicianbench
1010
skill_init: skillopt/envs/livemathematicianbench/skills/initial.md
1111
split_mode: split_dir
12-
split_dir: data/ablation_splits/livemathematicianbench/2-1-7_seed42
12+
split_dir: data/livemathematicianbench_split
1313
data_path: ""
1414
split_output_dir: ""
1515
max_turns: 1

‎data/README.md‎

Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
# Data Manifests
2+
3+
This directory releases lightweight split manifests for the SkillOpt paper
4+
splits. These manifests are not full runnable benchmark payloads. To evaluate a
5+
benchmark, first materialize the full examples from the raw data source when
6+
needed, then point `--split_dir` at the split directory listed below.
7+
8+
In this README, "coverage" describes which part of the upstream benchmark the
9+
manifest references. It does not mean the released manifest directory contains
10+
the full runnable examples.
11+
12+
## Layout
13+
14+
Every released manifest directory uses the same file layout:
15+
16+
```text
17+
data/<benchmark>_<manifest_type>/
18+
|-- split_manifest.json
19+
|-- train/items.json
20+
|-- val/items.json
21+
`-- test/items.json
22+
```
23+
24+
`split_manifest.json` records source metadata, split counts, and item fields.
25+
Each `items.json` contains only stable IDs or source-path hints.
26+
27+
## Released Splits
28+
29+
| Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
30+
|---|---|---:|---|---|---|
31+
| `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
32+
| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
33+
| `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
34+
| `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
35+
| `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |
36+
| `alfworld_path_split/` | ALFWorld | 39 / 18 / 134 | ALFWorld `json_2.1.1` paths | [alfworld/alfworld](https://github.com/alfworld/alfworld) | `data/alfworld_path_split` |
37+
38+
Counts are ordered as train / val / test.
39+
40+
## Direct Use
41+
42+
Only `alfworld_path_split/` can be used directly as `--split_dir` from this
43+
release, because the ALFWorld loader reads `gamefile` and `task_type` from the
44+
split items.
45+
46+
This does not mean the ALFWorld raw data is included. You still need to
47+
download ALFWorld separately with `alfworld-download` and set `$ALFWORLD_DATA`
48+
to the data root containing `json_2.1.1`.
49+
50+
The other manifest directories are lookup manifests. They intentionally omit
51+
full example fields such as questions, answers, contexts, images, or task
52+
instructions. Materialize those benchmarks into the `split_dir` paths listed
53+
above before running SkillOpt.
54+
55+
## Lookup Keys
56+
57+
The manifests are sufficient to locate the corresponding raw examples after
58+
the raw data has been downloaded or otherwise made available:
59+
60+
| Benchmark | Manifest lookup key |
61+
|---|---|
62+
| SearchQA | Match `items.json[].id` to the `key` field in `lucadiliello/searchqa`. |
63+
| LiveMathematicianBench | Open `source_file`, then match `no`; the manifest `id` is `<month>:<no>`. |
64+
| DocVQA | Match `questionId` within the official DocVQA `validation` split; `image_path` records the expected local image path. |
65+
| OfficeQA | Match `uid` in `officeqa_full.csv`; `source_files` and `source_docs` identify the supporting document. |
66+
| SpreadsheetBench | Match `id`; `spreadsheet_path` identifies the referenced spreadsheet directory. |
67+
| ALFWorld | Resolve `gamefile` relative to `$ALFWORLD_DATA`. |
68+
69+
## Manifest Item Examples
70+
71+
SearchQA:
72+
73+
```json
74+
{
75+
"id": "221c83e6630f4e7983da48fa28da1882"
76+
}
77+
```
78+
79+
LiveMathematicianBench:
80+
81+
```json
82+
{
83+
"id": "202602:22",
84+
"month": "202602",
85+
"no": 22,
86+
"paper_link": "http://arxiv.org/abs/2602.10700v1",
87+
"source_file": "data/202602/qa_202602_final.json"
88+
}
89+
```
90+
91+
DocVQA:
92+
93+
```json
94+
{
95+
"id": "50877",
96+
"questionId": "50877",
97+
"docId": "14724",
98+
"image_path": "data/docvqa_images/q50877_d14724.png",
99+
"source_split": "validation"
100+
}
101+
```
102+
103+
OfficeQA:
104+
105+
```json
106+
{
107+
"id": "UID0002",
108+
"uid": "UID0002",
109+
"category": "easy",
110+
"source_files": "treasury_bulletin_1944_01.txt"
111+
}
112+
```
113+
114+
SpreadsheetBench:
115+
116+
```json
117+
{
118+
"id": "32438",
119+
"spreadsheet_path": "spreadsheet/32438",
120+
"instruction_type": "Cell-Level Manipulation"
121+
}
122+
```
123+
124+
ALFWorld:
125+
126+
```json
127+
{
128+
"id": "train:0000",
129+
"gamefile": "json_2.1.1/train/.../game.tw-pddl",
130+
"task_type": "look_at_obj_in_light"
131+
}
132+
```
133+
134+
## Benchmark Notes
135+
136+
### SearchQA
137+
138+
`searchqa_id_split/` is an ID-only manifest. Each released `id` exactly matches
139+
the `key` field in `lucadiliello/searchqa`.
140+
141+
Materialized examples must include the fields consumed by the SearchQA
142+
environment, including:
143+
144+
```text
145+
question
146+
context
147+
answers
148+
```
149+
150+
### LiveMathematicianBench
151+
152+
`livemathematicianbench_id_split/` was generated from these raw files:
153+
154+
```text
155+
data/202511/qa_202511_final.json
156+
data/202512/qa_202512_final.json
157+
data/202601/qa_202601_final.json
158+
data/202602/qa_202602_final.json
159+
```
160+
161+
The manifest stores IDs in the loader format:
162+
163+
```text
164+
<month>:<no>
165+
```
166+
167+
Materialized examples must include:
168+
169+
```text
170+
question
171+
choices
172+
correct_choice
173+
theorem_type
174+
theorem
175+
sketch
176+
paper_link
177+
```
178+
179+
### DocVQA
180+
181+
`docvqa_id_split/` records `docvqa_validation_10pct`: a 10% subset sampled from
182+
the official DocVQA `validation` split.
183+
184+
```text
185+
source_split: validation
186+
docvqa_validation_10pct: train=107, val=53, test=374
187+
```
188+
189+
Each manifest item contains question/document IDs plus image location metadata.
190+
Materialized examples must provide `question`, `answer` or `ground_truth`, and
191+
an `image_path` that resolves locally.
192+
193+
### OfficeQA
194+
195+
`officeqa_id_split/` records the split over OfficeQA Full
196+
(`officeqa_full.csv`). The official OfficeQA CSVs are gated on Hugging Face, so
197+
materialization requires authorized access.
198+
199+
Each manifest item contains `uid`, `category`, `source_files`, and
200+
`source_docs` hints. Materialized examples must include `question` and
201+
`ground_truth` or `answer`.
202+
203+
### SpreadsheetBench
204+
205+
`spreadsheetbench_id_split/` records the split over SpreadsheetBench Verified
206+
400, from `spreadsheetbench_verified_400.tar.gz`.
207+
208+
Each manifest item contains task identity metadata such as `id`,
209+
`spreadsheet_path`, and `instruction_type`. Materialization must also place the
210+
referenced spreadsheet directories at:
211+
212+
```text
213+
data/spreadsheetbench_verified_400
214+
```
215+
216+
### ALFWorld
217+
218+
`alfworld_path_split/` records `gamefile` paths relative to `$ALFWORLD_DATA`.
219+
The source payload is `json_2.1.1`, which must be downloaded separately with
220+
`alfworld-download`.
221+
222+
This manifest can be used directly as `--split_dir` after `$ALFWORLD_DATA`
223+
points to the local ALFWorld data root containing `json_2.1.1`.

‎data/alfworld_path_split/split_manifest.json‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
{
2+
"benchmark": "ALFWorld",
3+
"manifest_type": "path_split",
4+
"source_repo": "alfworld/alfworld",
5+
"source_repo_type": "repository",
6+
"source_url": "https://github.com/alfworld/alfworld",
7+
"source_file": "json_2.1.1",
8+
"source_method": "generated by alfworld-download",
9+
"source_split_files": [
10+
"split_train.json",
11+
"split_val.json",
12+
"split_test.json"
13+
],
14+
"counts": {
15+
"train": 39,
16+
"val": 18,
17+
"test": 134
18+
},
19+
"item_fields": [
20+
"id",
21+
"gamefile",
22+
"task_type"
23+
],
24+
"path_root": "$ALFWORLD_DATA",
25+
"notes": [
26+
"This is a path manifest, not the ALFWorld game payload.",
27+
"The gamefile field is relative to ALFWORLD_DATA and must be expanded before direct use as split_dir data."
28+
]
29+
}

0 commit comments

Comments
 (0)