I want to use plyr on a large data.frame but make sure that I don't accidentally create copies of the dataset which could cause RAM to blow up.
I understand that when using plyr serially in non-parallel mode, it is careful to not make copies of the data.frame. From the original paper:
It is more memory-efficient than the naive split-apply-combine approach because plyr is careful not to make an extra copy of the data in the split step.
However, if using plyr in parallel mode, is the entire data.frame copied at each core? or is only 1 copy used for the entire analysis?
I want to use
plyron a largedata.framebut make sure that I don't accidentally create copies of the dataset which could cause RAM to blow up.I understand that when using
plyrserially in non-parallel mode, it is careful to not make copies of thedata.frame. From the original paper:However, if using
plyrin parallel mode, is the entiredata.framecopied at each core? or is only 1 copy used for the entire analysis?