Option to Delete Raw at the end of Brainset Pipelines

[AlexandreAndr]

Most people would not need to keep a version of the raw data or at least it could be nice to have the option to not save it. Especially when the dataset are really big and people have some storage constrains.
From that reflexion 2 questions come to my mind:

• Should it be a flag that is by default true or false?
• Should we delete them dynamically, like at the end of a file processing or all at the end?

[AlexandreAndr]

cc @vinamarora8 @milosobral

[milosobral]

To answer both questions:

• I think it's an option that should by default be False. I think keeping the raw data can be practical if you ever want to reprocess after modifications of the script without having to redownload the data from the source
• Deleting at the end is probably simpler but would be less storage efficient.

[AlexandreAndr]

I agree with the point 2, I can try to delete at the end to maximize the efficiency.
For point 1, after think about, the default can be determined based on who we consider as the default user:

• If it is someone developping the pipeline, I agree we should keep raw to iterate faster
• If it is a user that just wants to use the dataset and not develop on the pipeline, I am not sure they would need to have the raw data
  What do you think?

[AlexandreAndr]

Another point for keeping the default as false, it is to not have a change on the current default behavior

[milosobral]

I think the "default" user should definitely be # 2, i.e. not a developper of the pipeline, but I still think that we should keep the default to be to keep the raw data cached. Most libraries that do somewhat similar stuff like pytorch or huggingface datasets seem to keep the raw/compressed versions so you don't have to redownload by default. I also agree with your point that it also mean one less thing in the behavior that changes.

[divyansha1115]

I’ve been using Brainsets for large-scale processing and ended up adding a keep_raw flag for this reason.

In practice, I only keep the raw data while debugging or iterating on a pipeline, since it avoids having to download the same files again. Once the pipeline is stable, I prefer the raw data to be deleted immediately after each session has been processed.

For large datasets, keeping all raw files until the end can consume a significant amount of storage. I’ve had pipelines crash because the machine ran out of disk space while raw data was accumulating.

Storage can also be expensive when working with large scale data, so I think retaining raw data should be an explicit choice made by the user rather than the default behavior.

[AlexandreAndr]

I agree with both comments!
I think we can have some difficulty because we have the same interface for user or dev of a brainset pipeline.
I also like the idea of a compressed version of the data (maybe a following feature). The usecase, I had for IBL brainwidemap dataset that include a 1T for the raw data

[AlexandreAndr]

@vinamarora8 do not hesitate to comment here to