Skip to content

Fix GCS signature mismatch#3880

Open
akselbor wants to merge 4 commits intoPeerDB-io:mainfrom
economicmind:fix-gcs-signature-mismatch
Open

Fix GCS signature mismatch#3880
akselbor wants to merge 4 commits intoPeerDB-io:mainfrom
economicmind:fix-gcs-signature-mismatch

Conversation

@akselbor
Copy link
Copy Markdown

@akselbor akselbor commented Jan 31, 2026

Issue

Previously, the old "GCS workaround" worked fine for smaller payloads, but would fail for larger batches. In my testing, it appears to trigger once the requests become large enough to trigger multipart HTTP uploads. See #3877 for details.

The old workaround re‑signed the request too late in the pipeline, so it never fixed the cases where the SDK had already wrapped the body with streaming/chunk signatures for large/multipart uploads.

  • The previous logic in peerdb/flow/connectors/utils/aws.go used a custom http.RoundTripper (RecalculateV4Signature) to delete Accept-Encoding, re‑sign, then put the header back.
  • That happens after the AWS SDK has already signed the request and (for large bodies/multipart/streaming) wrapped the body with checksums/chunk signatures derived from the original signature.
  • Re‑signing only updates Authorization (and maybe X‑Amz‑Date), but does not update those body‑level signatures. For large/multipart uploads, that mismatch shows up as SignatureDoesNotMatch.

Changes

  • For GCS endpoints, force RequestChecksumCalculation/ResponseChecksumValidation = when_required on the S3 client.
  • For the transfer manager uploader, force RequestChecksumCalculation = when_required when targeting GCS.

@akselbor akselbor had a problem deploying to external-contributor January 31, 2026 13:56 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor January 31, 2026 13:56 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor January 31, 2026 13:56 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor January 31, 2026 13:56 — with GitHub Actions Failure
@CLAassistant
Copy link
Copy Markdown

CLAassistant commented Jan 31, 2026

CLA assistant check
All committers have signed the CLA.

@akselbor akselbor force-pushed the fix-gcs-signature-mismatch branch from 11989e3 to 3d9078b Compare January 31, 2026 14:13
@akselbor akselbor temporarily deployed to external-contributor January 31, 2026 14:13 — with GitHub Actions Inactive
@akselbor akselbor temporarily deployed to external-contributor January 31, 2026 14:13 — with GitHub Actions Inactive
@akselbor akselbor temporarily deployed to external-contributor January 31, 2026 14:13 — with GitHub Actions Inactive
@akselbor akselbor temporarily deployed to external-contributor January 31, 2026 14:13 — with GitHub Actions Inactive
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:24 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:24 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:24 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:24 — with GitHub Actions Failure
@serprex serprex requested a review from jgao54 March 9, 2026 16:29
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:55 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:55 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:55 — with GitHub Actions Failure
@akselbor akselbor had a problem deploying to external-contributor March 9, 2026 16:55 — with GitHub Actions Failure
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants