Skip to content

fix: sync command region handling and bucket region auto-detection#858

Open
gchait wants to merge 1 commit intopeak:masterfrom
gchait:fix-region-detection
Open

fix: sync command region handling and bucket region auto-detection#858
gchait wants to merge 1 commit intopeak:masterfrom
gchait:fix-region-detection

Conversation

@gchait
Copy link
Copy Markdown

@gchait gchait commented Feb 6, 2026

Fix: Sync command region handling and auto-detection

Problem

This PR fixes two related issues with region handling in s5cmd:

  1. Sync command doesn't respect --source-region and --destination-region flags: While the cp command correctly applies these flags, the sync command ignores them, causing operations to fail when syncing between buckets in different regions.

  2. Region auto-detection fails when AWS profile has a region configured: When a profile specifies a region, the current code skips bucket region auto-detection. This causes signature mismatch errors when the actual bucket is in a different region than the profile's configured region.

Solution

1. Apply region flags in sync command (command/sync.go)

Modified getSourceAndDestinationObjects() to create separate storage option objects for source and destination clients, applying the respective region flags before creating the clients. This matches the pattern already used in the cp command.

// Create source client with source region
srcOpts := s.storageOpts
if s.srcRegion != "" {
    srcOpts.SetRegion(s.srcRegion)
}
sourceClient, err := storage.NewClient(ctx, srcurl, srcOpts)

// Create destination client with destination region
dstOpts := s.storageOpts
if s.dstRegion != "" {
    dstOpts.SetRegion(s.dstRegion)
}
destClient, err := storage.NewClient(ctx, dsturl, dstOpts)

2. Always auto-detect bucket regions (storage/s3.go)

Modified setSessionRegion() to always perform bucket region auto-detection when a bucket is specified, regardless of whether a region is configured in the profile. This makes s5cmd's behavior consistent with AWS CLI.

Changes:

  • Always call GetBucketRegion() when a bucket name is provided
  • Fall back to configured region only when auto-detection fails
  • Only skip auto-detection when no bucket is specified (bucket-less operations)

This approach aligns with AWS CLI's "it just works" philosophy, where cross-region operations work transparently.

Notes

The extra HEAD request for bucket region auto-detection is a minor overhead (one per bucket per session) that enables seamless cross-region operations, matching user expectations from AWS CLI.

Related Issues

Fixes issues where:

  • s5cmd sync --source-region us-east-1 --destination-region eu-west-1 's3://bucket1/*' s3://bucket2/ would fail
  • Using AWS profiles with region configurations would cause signature mismatches for buckets in different regions

- Apply --source-region and --destination-region flags in sync command
  * Modified getSourceAndDestinationObjects() to create separate storage
    options for source and destination clients
  * Matches the pattern already used in cp command

- Always auto-detect bucket region when bucket is specified
  * Modified setSessionRegion() to call GetBucketRegion() regardless of
    whether a region is configured in the AWS profile
  * Falls back to configured region only when auto-detection fails
  * Makes behavior consistent with AWS CLI

Fixes:
- sync command ignoring --source-region and --destination-region flags
- signature mismatch errors when AWS profile has a region configured
  but bucket is in a different region
@gchait gchait requested a review from a team as a code owner February 6, 2026 13:30
@gchait gchait requested review from denizsurmeli and ilkinulas and removed request for a team February 6, 2026 13:30
@gchait
Copy link
Copy Markdown
Author

gchait commented Feb 6, 2026

Regarding the failing CI, I considered adding public.ecr.aws/ to your minio image ref in the workflow (bitnami repo images are removed from Docker Hub but not from ECR).
But, there are already 2 other PRs with fixes for this.

terryrankine added a commit to terryrankine/s5cmd that referenced this pull request Mar 18, 2026
The sync command was ignoring --source-region and --destination-region
flags because it used the same storageOpts for both clients. Now creates
separate option copies with region overrides, matching the cp command's
existing pattern. (peak#858)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant