Move the bucket versioning request into S3Core - #1098
Merged
Merged
Conversation
#1084 added a GetBucketVersioning request built in S3FileSystem and a call to the private S3Core._is_directory_bucket(). As agreed on #1086, add S3Core.get_bucket_versioning(), which returns the versioning status or None, and the public S3Path.is_directory_bucket property, which also replaces the private helper in plan_multipart_copy(). The aio mv tests that mocked only the sync filesystem's _call now share the mock with its core, as the other tests do, so that they keep observing the request. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
laughingman7743
commented
Oct 4, 2026
| for bucket in buckets | ||
| if self._call(self._client.get_bucket_versioning, Bucket=bucket).get("Status") | ||
| == "Enabled" | ||
| bucket for bucket in buckets if self.core.get_bucket_versioning(bucket) == "Enabled" |
Member
Author
There was a problem hiding this comment.
Self-review round one (implementation behavior): CLEAN
- Base
47fb6f31749f505a723c314b03ebf1ca3c5b8114, heade12d5d584c0f5b6f6ff7944585ca66941be18e1a. - Requests:
_move_pairs()sends the same GetBucketVersioning, once per distinct candidate bucket.- Before:
self._call(self._client.get_bucket_versioning, Bucket=...), which delegated toS3Core.call(). - Now:
S3Core.get_bucket_versioning(), which sendsself.call(self._client.get_bucket_versioning, Bucket=..., **params). - The client, the retry policy, the filtering of inherited
request_kwargsand the error translation are therefore unchanged.
- Before:
- Result: the comparison with
"Enabled"is unchanged;Noneand"Suspended"still mean the bucket is not versioning-enabled. - Directory buckets:
S3Path.is_directory_bucketapplies the same--x-s3suffix rule to the same bucket name at all three former call sites. These are the adapter's candidate filter and the two checks inplan_multipart_copy(). - Tests: the aio
mvtests now set_sync_fs._call = _sync_fs._core.call, as the other tests do.- With the request sent by the core, the old wiring made
test_mv_null_version_onto_keyfail. - It also made the directory-bucket test pass without observing the request.
- With the request sent by the core, the old wiring made
- Callers: no other code used
S3Core._is_directory_bucket()(repository grep). - Limitation: offline only at this head. A live run is in progress.
| FileNotFoundError: If the bucket does not exist. | ||
| """ | ||
| response = self.call(self._client.get_bucket_versioning, Bucket=bucket, **params) | ||
| return cast(str | None, response.get("Status")) |
Member
Author
There was a problem hiding this comment.
Self-review round two (claims, callers, operations): CLEAN
- Base
47fb6f31749f505a723c314b03ebf1ca3c5b8114, heade12d5d584c0f5b6f6ff7944585ca66941be18e1a. - I checked these claims:
- "
Nonefor a bucket whose versioning has never been enabled": GetBucketVersioning omitsStatusin that state, according to the botocore model docs and the S3 API reference. Stubber pins the empty response. - "A missing bucket raises
FileNotFoundError":S3ClientErrormapsNoSuchBucketto it, and Stubber pins this. - "
ExpectedBucketOwneris passed through andRequestPayeris filtered": the operation's input shape has onlyBucketandExpectedBucketOwner, measured with botocore 1.43.102. - "No change of requests": the existing sync
mvtests assertfs._call.assert_called_once_with(fs._client.get_bucket_versioning, Bucket="bucket")through the shared core mock, and they pass unchanged.
- "
- The docs sentence matches the docstring.
- Breaking-change check: the private staticmethod is removed. It was underscore-private and only the core used it, so no release note is needed.
laughingman7743
force-pushed
the
refactor/1086-bucket-versioning
branch
from
October 4, 2026 16:33
5c66f58 to
e12d5d5
Compare
laughingman7743
commented
Oct 4, 2026
| return not self.key or not self.key.strip("/") | ||
|
|
||
| @property | ||
| def is_directory_bucket(self) -> bool: |
Member
Author
There was a problem hiding this comment.
Independent review and /code-review at e12d5d584c0f5b6f6ff7944585ca66941be18e1a (base 47fb6f31749f505a723c314b03ebf1ca3c5b8114, detached snapshot, static only)
- Codex (
codex exec -s read-only): CLEAN. It covered the versioning request parameters, the status returns and error translation, the sync/aio move behavior and lookup counts, directory-bucket detection, the multipart-copy guards, the tests and the docs. - claude-fable-5-1 (
maxprofile, efforthigh): CLEAN. It confirmed that the request, the request count, the retry policy, the error translation and the== "Enabled"result are unchanged. It also confirmed that the four aio tests are the only ones that mocked_callalone. - /code-review: no regression. Nine design, test-precision and docs notes, none changed:
- The name
is_directory_bucketis kept. "Directory bucket" is the AWS term, and the docstring already says it concerns the bucket of the path. The maintainer decided this. - Docstring and docs additions about directory-bucket support and paragraph placement were judged unnecessary and reverted. Other core operations do not list S3 feature support either.
- The
str | Nonereturn was agreed on Finish the public S3Core API extraction from filesystem adapters #1086. - Not changed because they are pre-existing behavior or out of scope: sequential GetBucketVersioning per bucket,
--xa-s3access point aliases, and test assertions through the shared_callalias.
- The name
- Live AWS at
e12d5d584c0f5b6f6ff7944585ca66941be18e1a:pytest -n 8 tests/pyathena/filesystem/ tests/pyathena/s3fs/ tests/pyathena/aio/s3fs/→ 1235 passed, 10 skipped.
laughingman7743
marked this pull request as ready for review
October 4, 2026 16:37
laughingman7743
marked this pull request as draft
October 4, 2026 16:47
laughingman7743
marked this pull request as ready for review
October 4, 2026 18:47
This was referenced Oct 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
WHAT
This is the fourth and last implementation PR for #1086, following the addendum.
It moves the two pieces that #1084 added to
S3FileSystem._move_pairs()after the #1086 inventory was recorded.S3Core.get_bucket_versioning(bucket, **params) -> str | Nonesends one GetBucketVersioning request."Enabled","Suspended", orNonefor a bucket whose versioning has never been enabled.FileNotFoundError.S3Path.is_directory_bucketsays whether the path's bucket is a directory bucket (a name ending with--x-s3).S3Core._is_directory_bucket()staticmethod, whichplan_multipart_copy()also used._move_pairs()uses both.S3Core.call()as before.docs/filesystem.mddocuments both.mvtests that mocked only_sync_fs._callnow share the mock with_sync_fs._core.call, as the other tests do.test_mv_null_version_onto_keyfailing.test_mv_null_version_directory_bucket_does_not_read_bucket_statepass without observing anything.WHY
Refs #1086.
#1084 merged after the inventory, so these were the last S3 request and S3 rule built in the adapter.
TEST
Tested commit: e12d5d5.
just format,just lintandjust docs lintpass.uv run --env-file <placeholder env> pytest --noconftest -p no:rerunfailures -n 8 tests/pyathena/filesystem/47fb6f31: 958 passed, 164 failed, 9 skipped.is_directory_bucketand 4 forget_bucket_versioning.RequestPayerfiltered out andExpectedBucketOwnerpassed through;NoSuchBucket→FileNotFoundErrortranslation.mvnull-version tests pin the adapter's request sequence.uv run --env-file .env pytest -n 8 tests/pyathena/filesystem/ tests/pyathena/s3fs/ tests/pyathena/aio/s3fs/→ 1235 passed, 10 skipped. No other Test run overlapped.🤖 Generated with Claude Code