Skip to content

DATASETS.md

Latest commit

 

History

History
94 lines (68 loc) · 3.61 KB

DATASETS.md

File metadata and controls

94 lines (68 loc) · 3.61 KB
Raw

Dataset and Benchmark Preparation

This repository contains code and a tiny synthetic example. It does not redistribute source SFT pools, selected experimental subsets, benchmark files, model weights, embeddings, indexes, checkpoints, or logs.

Source pools used in the paper

The single-domain study uses two public SFT pools per domain:

Domain Source pool Official source
Math OpenR1-Math-220k https://huggingface.co/datasets/open-r1/OpenR1-Math-220k
Math Synthetic-Math Obtain the release identified by the paper and verify its current Dataset Card before use.
Medicine Medical-Reasoning Obtain the release identified by the paper and verify its current Dataset Card before use.
Medicine UltraMedical https://huggingface.co/datasets/TsinghuaC3I/UltraMedical
Code OSS-Instruct Obtain the release identified by the paper and verify its current Dataset Card before use.
Code Code-Alpaca https://github.com/sahil280114/codealpaca

The multi-domain study uses:

OpenHermes-2.5 targets math, code, and science. Tulu-3 targets math and code. No artificial Tulu-3 science partition is constructed.

Before downloading or redistributing any dataset, inspect its current Dataset Card, license, component-dataset terms, and access conditions. The repository's Apache-2.0 license applies only to project-authored code.

Normalize an SFT pool

Create one directory per source pool:

dataset/
`-- pool_name/
    |-- data_000.json
    |-- data_001.json
    `-- ...

Each JSON file must contain a list of records with instruction, optional input, output, and serialized text fields. Splitting a large pool across multiple files is supported.

Do not mix previous outputs such as filtered_*.json, cached embeddings, or selection directories into a fresh input directory.

Dataset-card summary and pool profile

Copy examples/dataset_card_profile.example.yaml and fill it with information from the official Dataset Card:

  • dataset ID and exact revision;
  • split name and Dataset Card URL;
  • domain coverage and languages;
  • construction procedure and intended SFT format;
  • any relevant usage or filtering notes.

Do not invent a revision that was not recorded. Running preprocessing with --dataset-card-profile combines this summary with deterministic statistics from the exact local split and writes profile.txt beside the source JSON.

Evaluation benchmarks

Benchmark files are not redistributed. Place each benchmark in a separate directory and pass the relevant directories through --test_data_dirs.

The paper uses the following benchmark families:

  • math: GSM8K, MATH, Minerva-Math, and the stated competition benchmark;
  • medicine: MedQA and MMLU-Medical;
  • code: LiveCodeBench, HumanEval, and BigCodeBench;
  • science: MMLU-STEM and GPQA-Diamond.

Evaluator dispatch is based on benchmark-directory names. Inspect finetune/test/evaluators/__init__.py before naming a local benchmark folder. Preserve each benchmark's official format, split, license, and evaluation policy.

Outputs that should remain untracked

The root .gitignore excludes common generated artifacts, including:

  • dataset/ and benchmark/;
  • embeddings and FAISS files;
  • logs, outputs, stage visualizations, and checkpoints;
  • model weight formats and NumPy caches;
  • populated config.yaml.

Review git status before every commit because an upstream dataset may create additional cache or metadata files not covered by the default patterns.