Skip to content

Question about parallelizing plyr #308

Description

@jarbet

I want to use plyr on a large data.frame but make sure that I don't accidentally create copies of the dataset which could cause RAM to blow up.

I understand that when using plyr serially in non-parallel mode, it is careful to not make copies of the data.frame. From the original paper:

It is more memory-efficient than the naive split-apply-combine approach because plyr is careful not to make an extra copy of the data in the split step.

However, if using plyr in parallel mode, is the entire data.frame copied at each core? or is only 1 copy used for the entire analysis?

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions