Add max delete option by grmrgecko · Pull Request #699 · peak/s5cmd

grmrgecko · 2024-02-07T20:02:19Z

In reviewing options to fix the issue where errors deleted all files in the destination, I thought that having a maximum delete option similar to what rysnc has would have also been an nice option. So here it is.

Sync test of test data set:

$ ./s5cmd --endpoint-url='https://objects.gec.im sync --no-follow-symlinks --delete 's3://test-jcoleman/*' ~/test/
cp s3://test-jcoleman/test-dir-1/test-file-13 /home/grmrgecko/test/test-dir-1/test-file-13
cp s3://test-jcoleman/test-dir-1/test-file-10 /home/grmrgecko/test/test-dir-1/test-file-10
cp s3://test-jcoleman/test-dir-1/test-file-14 /home/grmrgecko/test/test-dir-1/test-file-14
cp s3://test-jcoleman/test-dir-1/test-file-15 /home/grmrgecko/test/test-dir-1/test-file-15
cp s3://test-jcoleman/test-dir-1/test-file-11 /home/grmrgecko/test/test-dir-1/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-19 /home/grmrgecko/test/test-dir-1/test-file-19
cp s3://test-jcoleman/test-dir-1/test-file-22 /home/grmrgecko/test/test-dir-1/test-file-22
cp s3://test-jcoleman/test-dir-2/test-file-12 /home/grmrgecko/test/test-dir-2/test-file-12
cp s3://test-jcoleman/test-dir-1/test-file-6 /home/grmrgecko/test/test-dir-1/test-file-6
cp s3://test-jcoleman/test-dir-1/test-file-21 /home/grmrgecko/test/test-dir-1/test-file-21
cp s3://test-jcoleman/test-dir-2/test-file-11 /home/grmrgecko/test/test-dir-2/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-3 /home/grmrgecko/test/test-dir-1/test-file-3
cp s3://test-jcoleman/test-dir-1/test-file-9 /home/grmrgecko/test/test-dir-1/test-file-9
cp s3://test-jcoleman/test-dir-2/test-file-1 /home/grmrgecko/test/test-dir-2/test-file-1
cp s3://test-jcoleman/test-dir-1/test-file-20 /home/grmrgecko/test/test-dir-1/test-file-20
cp s3://test-jcoleman/test-dir-1/test-file-18 /home/grmrgecko/test/test-dir-1/test-file-18
cp s3://test-jcoleman/test-dir-1/test-file-16 /home/grmrgecko/test/test-dir-1/test-file-16
cp s3://test-jcoleman/test-dir-1/test-file-7 /home/grmrgecko/test/test-dir-1/test-file-7
cp s3://test-jcoleman/test-dir-2/test-file-15 /home/grmrgecko/test/test-dir-2/test-file-15
cp s3://test-jcoleman/test-dir-2/test-file-4 /home/grmrgecko/test/test-dir-2/test-file-4
cp s3://test-jcoleman/test-dir-2/test-file-13 /home/grmrgecko/test/test-dir-2/test-file-13
cp s3://test-jcoleman/test-dir-2/test-file-20 /home/grmrgecko/test/test-dir-2/test-file-20
cp s3://test-jcoleman/test-dir-2/test-file-16 /home/grmrgecko/test/test-dir-2/test-file-16
cp s3://test-jcoleman/test-dir-3/test-file-14 /home/grmrgecko/test/test-dir-3/test-file-14
cp s3://test-jcoleman/test-dir-2/test-file-8 /home/grmrgecko/test/test-dir-2/test-file-8
cp s3://test-jcoleman/test-dir-2/test-file-19 /home/grmrgecko/test/test-dir-2/test-file-19
cp s3://test-jcoleman/test-dir-3/test-file-15 /home/grmrgecko/test/test-dir-3/test-file-15
cp s3://test-jcoleman/test-dir-2/test-file-21 /home/grmrgecko/test/test-dir-2/test-file-21
cp s3://test-jcoleman/test-dir-3/test-file-5 /home/grmrgecko/test/test-dir-3/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-16 /home/grmrgecko/test/test-dir-3/test-file-16
cp s3://test-jcoleman/test-dir-2/test-file-3 /home/grmrgecko/test/test-dir-2/test-file-3
cp s3://test-jcoleman/test-dir-3/test-file-10 /home/grmrgecko/test/test-dir-3/test-file-10
cp s3://test-jcoleman/test-dir-2/test-file-7 /home/grmrgecko/test/test-dir-2/test-file-7
cp s3://test-jcoleman/test-dir-3/test-file-1 /home/grmrgecko/test/test-dir-3/test-file-1
cp s3://test-jcoleman/test-dir-2/test-file-6 /home/grmrgecko/test/test-dir-2/test-file-6
cp s3://test-jcoleman/test-dir-3/test-file-12 /home/grmrgecko/test/test-dir-3/test-file-12
cp s3://test-jcoleman/test-dir-3/test-file-3 /home/grmrgecko/test/test-dir-3/test-file-3
cp s3://test-jcoleman/test-file-3 /home/grmrgecko/test/test-file-3
cp s3://test-jcoleman/test-dir-3/test-file-22 /home/grmrgecko/test/test-dir-3/test-file-22
cp s3://test-jcoleman/test-dir-3/test-file-20 /home/grmrgecko/test/test-dir-3/test-file-20
cp s3://test-jcoleman/test-dir-3/test-file-2 /home/grmrgecko/test/test-dir-3/test-file-2
cp s3://test-jcoleman/test-dir-3/test-file-9 /home/grmrgecko/test/test-dir-3/test-file-9
cp s3://test-jcoleman/test-dir-3/test-file-18 /home/grmrgecko/test/test-dir-3/test-file-18
cp s3://test-jcoleman/test-dir-3/test-file-7 /home/grmrgecko/test/test-dir-3/test-file-7
cp s3://test-jcoleman/test-file-6 /home/grmrgecko/test/test-file-6
cp s3://test-jcoleman/test-dir-3/test-file-8 /home/grmrgecko/test/test-dir-3/test-file-8
cp s3://test-jcoleman/test-file-2 /home/grmrgecko/test/test-file-2
cp s3://test-jcoleman/test-dir-1/test-file-1 /home/grmrgecko/test/test-dir-1/test-file-1
cp s3://test-jcoleman/test-dir-2/test-file-17 /home/grmrgecko/test/test-dir-2/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-9 /home/grmrgecko/test/test-dir-2/test-file-9
cp s3://test-jcoleman/test-dir-3/test-file-19 /home/grmrgecko/test/test-dir-3/test-file-19
cp s3://test-jcoleman/test-dir-1/test-file-17 /home/grmrgecko/test/test-dir-1/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-22 /home/grmrgecko/test/test-dir-2/test-file-22
cp s3://test-jcoleman/test-file-4 /home/grmrgecko/test/test-file-4
cp s3://test-jcoleman/test-dir-1/test-file-5 /home/grmrgecko/test/test-dir-1/test-file-5
cp s3://test-jcoleman/test-dir-2/test-file-10 /home/grmrgecko/test/test-dir-2/test-file-10
cp s3://test-jcoleman/test-file-5 /home/grmrgecko/test/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-13 /home/grmrgecko/test/test-dir-3/test-file-13
cp s3://test-jcoleman/test-dir-2/test-file-14 /home/grmrgecko/test/test-dir-2/test-file-14
cp s3://test-jcoleman/test-dir-1/test-file-12 /home/grmrgecko/test/test-dir-1/test-file-12
cp s3://test-jcoleman/test-dir-2/test-file-18 /home/grmrgecko/test/test-dir-2/test-file-18
cp s3://test-jcoleman/test-dir-3/test-file-11 /home/grmrgecko/test/test-dir-3/test-file-11
cp s3://test-jcoleman/test-dir-1/test-file-8 /home/grmrgecko/test/test-dir-1/test-file-8
cp s3://test-jcoleman/test-dir-2/test-file-5 /home/grmrgecko/test/test-dir-2/test-file-5
cp s3://test-jcoleman/test-dir-3/test-file-17 /home/grmrgecko/test/test-dir-3/test-file-17
cp s3://test-jcoleman/test-dir-2/test-file-2 /home/grmrgecko/test/test-dir-2/test-file-2
cp s3://test-jcoleman/test-dir-1/test-file-2 /home/grmrgecko/test/test-dir-1/test-file-2
cp s3://test-jcoleman/test-dir-3/test-file-21 /home/grmrgecko/test/test-dir-3/test-file-21
cp s3://test-jcoleman/test-dir-3/test-file-4 /home/grmrgecko/test/test-dir-3/test-file-4
cp s3://test-jcoleman/test-dir-1/test-file-4 /home/grmrgecko/test/test-dir-1/test-file-4
cp s3://test-jcoleman/test-file-1 /home/grmrgecko/test/test-file-1
cp s3://test-jcoleman/test-dir-3/test-file-6 /home/grmrgecko/test/test-dir-3/test-file-6

Test of max-delete with a data set that removes a few directories:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete --max-delete=5 's3://test-empty-jcoleman/*' ~/test/
Not deleting due 44 being higher than maximum delete limit

Test showing delete works with a higher maximum:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete --max-delete=50 's3://test-empty-jcoleman/*' ~/test/           
rm /home/grmrgecko/test/test-dir-2/test-file-10
rm /home/grmrgecko/test/test-dir-2/test-file-14
rm /home/grmrgecko/test/test-dir-3/test-file-9
rm /home/grmrgecko/test/test-dir-2/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-15
rm /home/grmrgecko/test/test-dir-2/test-file-1
rm /home/grmrgecko/test/test-dir-2/test-file-16
rm /home/grmrgecko/test/test-dir-2/test-file-13
rm /home/grmrgecko/test/test-dir-2/test-file-4
rm /home/grmrgecko/test/test-dir-3/test-file-13
rm /home/grmrgecko/test/test-dir-2/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-8
rm /home/grmrgecko/test/test-dir-2/test-file-18
rm /home/grmrgecko/test/test-dir-3/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-9
rm /home/grmrgecko/test/test-dir-3/test-file-7
rm /home/grmrgecko/test/test-dir-3/test-file-15
rm /home/grmrgecko/test/test-dir-2/test-file-7
rm /home/grmrgecko/test/test-dir-2/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-8
rm /home/grmrgecko/test/test-dir-3/test-file-1
rm /home/grmrgecko/test/test-dir-3/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-3
rm /home/grmrgecko/test/test-dir-3/test-file-16
rm /home/grmrgecko/test/test-dir-3/test-file-11
rm /home/grmrgecko/test/test-dir-3/test-file-10
rm /home/grmrgecko/test/test-dir-3/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-4
rm /home/grmrgecko/test/test-dir-2/test-file-2
rm /home/grmrgecko/test/test-dir-2/test-file-21
rm /home/grmrgecko/test/test-dir-3/test-file-18
rm /home/grmrgecko/test/test-dir-2/test-file-20
rm /home/grmrgecko/test/test-dir-3/test-file-2
rm /home/grmrgecko/test/test-dir-3/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-3

Test showing not defining option will default to the negative and delete files:

$ ./s5cmd --endpoint-url='https://objects.gec.im' sync --no-follow-symlinks --delete 's3://test-empty-jcoleman/*' ~/test/                  
rm /home/grmrgecko/test/test-dir-2/test-file-10
rm /home/grmrgecko/test/test-dir-2/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-1
rm /home/grmrgecko/test/test-dir-3/test-file-9
rm /home/grmrgecko/test/test-dir-2/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-2
rm /home/grmrgecko/test/test-dir-3/test-file-5
rm /home/grmrgecko/test/test-dir-3/test-file-11
rm /home/grmrgecko/test/test-dir-2/test-file-13
rm /home/grmrgecko/test/test-dir-3/test-file-6
rm /home/grmrgecko/test/test-dir-3/test-file-10
rm /home/grmrgecko/test/test-dir-3/test-file-12
rm /home/grmrgecko/test/test-dir-3/test-file-8
rm /home/grmrgecko/test/test-dir-3/test-file-7
rm /home/grmrgecko/test/test-dir-2/test-file-15
rm /home/grmrgecko/test/test-dir-3/test-file-13
rm /home/grmrgecko/test/test-dir-3/test-file-4
rm /home/grmrgecko/test/test-dir-3/test-file-22
rm /home/grmrgecko/test/test-dir-3/test-file-3
rm /home/grmrgecko/test/test-dir-2/test-file-16
rm /home/grmrgecko/test/test-dir-3/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-18
rm /home/grmrgecko/test/test-dir-3/test-file-15
rm /home/grmrgecko/test/test-dir-3/test-file-14
rm /home/grmrgecko/test/test-dir-2/test-file-17
rm /home/grmrgecko/test/test-dir-3/test-file-19
rm /home/grmrgecko/test/test-dir-3/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-3
rm /home/grmrgecko/test/test-dir-3/test-file-1
rm /home/grmrgecko/test/test-dir-2/test-file-8
rm /home/grmrgecko/test/test-dir-2/test-file-2
rm /home/grmrgecko/test/test-dir-2/test-file-9
rm /home/grmrgecko/test/test-dir-3/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-20
rm /home/grmrgecko/test/test-dir-2/test-file-18
rm /home/grmrgecko/test/test-dir-2/test-file-19
rm /home/grmrgecko/test/test-dir-2/test-file-21
rm /home/grmrgecko/test/test-dir-2/test-file-22
rm /home/grmrgecko/test/test-dir-2/test-file-6
rm /home/grmrgecko/test/test-dir-2/test-file-7
rm /home/grmrgecko/test/test-dir-3/test-file-16
rm /home/grmrgecko/test/test-dir-2/test-file-5
rm /home/grmrgecko/test/test-dir-2/test-file-4

ilkinulas · 2024-07-04T10:25:48Z

Hello! If you’d still like this to be merged, could you please:

Rebase your branch
Add a testcase to demonstrate that your implementation works

Test three boundary conditions: deletion proceeds when under the limit, and deletion is skipped when file count meets or exceeds the limit.

grmrgecko · 2026-02-13T05:12:04Z

Rebase is done.

grmrgecko requested a review from a team as a code owner February 7, 2024 20:02

grmrgecko requested review from ilkinulas and sonmezonur and removed request for a team February 7, 2024 20:02

grmrgecko mentioned this pull request Feb 7, 2024

Source error with sync delete all dest files #695

Open

ilkinulas added the sync label Jun 23, 2024

grmrgecko added 2 commits February 12, 2026 23:02

Add max delete option

97f72ad

Add e2e tests for sync --max-delete flag

0bfeb01

Test three boundary conditions: deletion proceeds when under the limit, and deletion is skipped when file count meets or exceeds the limit.

grmrgecko force-pushed the max_delete branch from c511d2f to 0bfeb01 Compare February 13, 2026 05:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add max delete option#699

Add max delete option#699
grmrgecko wants to merge 2 commits intopeak:masterfrom
grmrgecko:max_delete

grmrgecko commented Feb 7, 2024 •

edited

Loading

Uh oh!

ilkinulas commented Jul 4, 2024

Uh oh!

grmrgecko commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

grmrgecko commented Feb 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ilkinulas commented Jul 4, 2024

Uh oh!

grmrgecko commented Feb 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grmrgecko commented Feb 7, 2024 •

edited

Loading