Data Prepare

[XinDongol]

I am following

python setup/download_prepare_hf_data.py fineweb_edu 32 --data_dir ./data --seed 42 --nchunks 32

fineweb-edu is about 10TB.

The above commend write the shuffled file with speed of 10G/h which means it will take forever to finish the shuffling.

Any suggestion on this?

Thanks.

[thib-s]

Hi, I got into a similar issue and found out that, in my case, the issue came from the I/O read/write on a slow disk (I set the --data_dir to a directory on a drive over the network)

I fixed this by:

• setting data_dir to a local SSD/HDD (and synchronizing the folder on a network disk afterward)
• terashuf uses a temporary dir set by default to /tmp, which can sometimes be slow. You can set this location by editing lines 122-125 of the script to something like this:

    # Set up environment variables
    os.environ["MEMORY"] = f"{memory}"
    os.environ["SEED"] = f"{seed}"
    os.environ["TMPDIR"] = "some/fast/local/dir"

I hope this helps!