Merge pull request #176 from mvanhorn/fix/158-sync-livemath-split-metadata

fix: sync LiveMath split metadata

Author: Yif-Yang

.osc-metadata/sync.json

@@ -0,0 +1,5 @@
+{
+  "fork_synced_at": "2026-07-26T09:38:34.869801+00:00",
+  "commits_behind_before_sync": 210,
+  "action_taken": "synced"
+}

data/README.md

@@ -29,7 +29,7 @@ Each `items.json` contains only stable IDs or source-path hints.
 | Manifest directory | Benchmark | Counts | Coverage | Raw data source | `split_dir` |
 |---|---|---:|---|---|---|
 | `searchqa_id_split/` | SearchQA | 400 / 200 / 1400 | Official HF dataset IDs | [lucadiliello/searchqa](https://huggingface.co/datasets/lucadiliello/searchqa) | `data/searchqa_split` |
-| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 18 / 124 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
+| `livemathematicianbench_id_split/` | LiveMathematicianBench | 35 / 17 / 125 | Four official monthly files | [LiveMathematicianBench/LiveMathematicianBench](https://huggingface.co/datasets/LiveMathematicianBench/LiveMathematicianBench) | `data/livemathematicianbench_split` |
 | `docvqa_id_split/` | DocVQA | 107 / 53 / 374 | 10% subset of validation | [lmms-lab/DocVQA](https://huggingface.co/datasets/lmms-lab/DocVQA) | `data/docvqa/splits` |
 | `officeqa_id_split/` | OfficeQA | 50 / 24 / 172 | OfficeQA Full | [databricks/officeqa](https://huggingface.co/datasets/databricks/officeqa) | `data/officeqa_split` |
 | `spreadsheetbench_id_split/` | SpreadsheetBench | 80 / 40 / 280 | SpreadsheetBench Verified 400 | [KAKA22/SpreadsheetBench](https://huggingface.co/datasets/KAKA22/SpreadsheetBench) | `data/spreadsheetbench_split` |

data/livemathematicianbench_id_split/split_manifest.json

@@ -16,8 +16,8 @@
   "split_seed": 42,
   "counts": {
     "train": 35,
-    "val": 18,
-    "test": 124
+    "val": 17,
+    "test": 125
   },
   "item_fields": [
     "id",

tests/test_data_manifests.py

@@ -0,0 +1,23 @@
+import json
+from pathlib import Path
+
+from skillopt.datasets.base import SPLIT_NAMES
+
+DATA_DIR = Path(__file__).resolve().parents[1] / "data"
+
+
+def test_split_manifest_counts_match_item_files() -> None:
+    manifest_paths = sorted(DATA_DIR.glob("*/split_manifest.json"))
+    assert manifest_paths, f"No split manifests found under {DATA_DIR}"
+
+    for manifest_path in manifest_paths:
+        manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
+
+        for split_name in SPLIT_NAMES:
+            declared_count = manifest["counts"][split_name]
+            items_path = manifest_path.parent / split_name / "items.json"
+            actual_count = len(json.loads(items_path.read_text(encoding="utf-8")))
+
+            assert declared_count == actual_count, (
+                f"{manifest_path} split {split_name!r}: declared count {declared_count}, actual count {actual_count}"
+            )