Skip to content

shuf with interval is very inefficient and slow #109

Description

@xelan

Apparently, the implementation of the shuf command requires a lot of memory, as it creates a "virtual line" for each number in the range. For larger numbers (but well below INT_MAX) the command crashes with an "out of memory" error:

# about half of 32-bit INT_MAX
$ shuf -i 1-2222222222 -n 1
shuf: out of memory

For smaller numbers, the performance is worse than the GNU shuf binary by many orders of magnitude.

busybox shuf:

$ time shuf -i 1-99999999 -n 1
28235271

real	0m2.087s
user	0m1.588s
sys	0m0.424s

GNU shuf:

$ time shuf -i 1-99999999 -n 1
13884991

real	0m0,001s
user	0m0,000s
sys	0m0,001s

Thanks for investigating on this issue!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions