Repository navigation
Conversation
The S3 backend's `region` identifies the bucket; the region to provision
into is passed separately as `-var=region=`. `_tofu_init` sent `app.region`
for both, so any deployment whose region differed from its state bucket's
failed init with:
Failed to get existing workspaces: operation error S3: ListObjectsV2,
https response error StatusCode: 301 ... api error PermanentRedirect
OpenTofu's S3 backend does not follow S3's region redirect. The AWS CLI
does, which makes this easy to misdiagnose as working.
This became reachable in 0.3.0, when `app.region` started actually driving
the deployment. The bucket name is derived from the account ID and S3 names
are global, so a bucket an earlier deployment created in another region is
reused by name and silently mismatches.
Ask S3 where the bucket is via `get_bucket_location`, falling back to
`app.region` when the lookup fails — no worse than the previous assumption.
`_tofu_destroy` shares `_tofu_init`, so destroy is fixed too.
Found deploying to eu-west-1 against a us-west-2 state bucket.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014bWRfnmvBt12hWFk5FwAg3
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The S3 backend's
regionidentifies the state bucket. The region to provision into is passed separately as-var=region=._tofu_initsentcfg.app.regionfor both, so any deployment whose region differs from its state bucket's region fails at init:OpenTofu's S3 backend does not follow S3's region redirect. The AWS CLI does, which makes this easy to misdiagnose — a
head-bucket/lsagainst the bucket with the "wrong" region succeeds, so the config looks fine right up untiltofu init.This became reachable in 0.3.0, when
app.regionstarted actually driving the deployment (before that everything landed inus-west-2regardless). The bucket name is derived from the account ID and S3 bucket names are global, so a bucket an earlier deployment created in another region is reused by name and silently mismatches.Fix
Ask S3 where the bucket actually is via
get_bucket_location, and send that as the backend region. On any lookup failure it falls back tocfg.app.region— no worse than the assumption it replaces, so a caller withouts3:GetBucketLocationis unaffected._tofu_destroyshares_tofu_init, so destroy is fixed too.Testing
Four new tests in
TestResolveBucketRegion/TestTofuInitBackendRegion: bucket location is used, a nullLocationConstraintresolves tous-east-1, lookup failure falls back, and_tofu_initsends the bucket's region rather thanapp.region.packages/cli: 835 passed, 1 failed, and that failure (test_docker_isolation.py::test_explicit_subprocess_patch_overrides_the_autouse_guard) is pre-existing and environmental — it fails identically on cleanmainon this machine because Docker isn't installed.ruff check packages/cliis clean.Found while deploying to
eu-west-1against aus-west-2state bucket; the deployment is now live and healthy with the patch applied.Note: the allocator has the same bug, and it needs a template change too
packages/allocator/src/lablink_allocator_service/main.py(~line 360) builds the client-VM workspace's backend the same way:It fails identically, and worse — it crashes allocator startup, so the service never becomes healthy (
Flask process exited before becoming ready). I have not fixed it here, because the same fix alone wouldn't work: the allocator calls AWS with its instance role, and thes3_backend_docpolicy inlablink-templategrantss3:ListBucketand object actions but nots3:GetBucketLocation. The lookup would fail, hit the fallback, and reproduce the bug. Fixing it properly means a coordinated change across both repos, so it seemed better as a separate PR than bundled here.Workaround in the meantime: point
bucket_nameat a bucket inapp.region, which the allocator honors (unlike the CLI, which derives its own name and ignorescfg.bucket_name— arguably its own inconsistency).🤖 Generated with Claude Code
https://claude.ai/code/session_014bWRfnmvBt12hWFk5FwAg3