Skip to content

8.0 fixes: purge after collections delete, --local in S3 mode, and duplicate variants - #113

Open
harmony7 wants to merge 3 commits into
mainfrom
fix/s3-mode-fixes
Open

harmony7 wants to merge 3 commits into
mainfrom
fix/s3-mode-fixes

Conversation

@harmony7

@harmony7 harmony7 commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Summary

Three fixes for 8.0, before the first stable release with S3-compatible storage. Each fix is one commit.

This PR replaces #110. Its commit is the first commit here, with no changes.

Closes #111.
Closes #112.

PR Guide:
review-113.pdf

1. Purge the collection after collections delete (from #110)

After a collection is deleted, the service can continue to serve its cached responses until they expire. collections delete now purges the surrogate key <publishId>-<collection>, the same as publish-content does after a publish.

  • Same options and rules as publish-content: --fastly-service-id (then FASTLY_SERVICE_ID, then service_id in fastly.toml), --purge-environment, and the API token. If the CLI finds a Service ID but no API token, it stops before it deletes anything. If it finds no Service ID, or with --local, it skips the purge. If the purge fails, it shows a warning, and the command completes.
  • One purge function: purgeCollection() in src/cli/util/purge.ts. Both commands use it.
  • Users who upgrade see a change: if collections delete finds a Service ID, it now needs an API token that can purge that service. MIGRATING.md says this.

2. --local in S3 mode stops with an error (#111)

In S3 mode, the commands ignored --local for storage. publish-content --local uploaded to the bucket, and collections delete --local and clean --local deleted from it. There is no local simulation of S3 storage.

  • The S3 storage provider builder now throws an error if --local is set, before it prints or connects to anything. All six commands that accept --local load the provider through this builder, so they all stop with exit code 1: --local is available only in KV Store mode. In S3 mode, the commands always use the bucket in static-publish.rc.js.
  • The generated S3 app does not use --local (it has only s3:publish), so it does not change.
  • In S3 mode, --local had one effect before: publish-content skipped the purge. The CHANGELOG tells users who used it for that what to do.
  • --help and the README now tell that --local is for KV Store mode only.

3. Prepare each variant of the same content one time (#112)

publish-content scans up to 16 files at the same time. Files with the same content checked the variant map before the first of them set it. Thus each of these files compressed the content again (mostly visible for Brotli), and added the same keys to the upload batch (in S3 mode, one PutObject for each duplicate). Several files could also write the same variant file at the same time.

  • The new OncePerKey helper (src/cli/util/variants.ts) keeps one promise for each variant storage key. The first file starts the work and stores the promise before its first await. The other files await the same promise. Only the first file adds the key to the batch.
  • The uploaded content does not change.
  • publish-content --local now shows the number of variants to upload. Before, local mode showed no count, because it does not list the keys in storage.

Tests

  • npm test: 77 pass. New: unit tests for purgeCollection() (from Purge the collection after collections delete #110) and for OncePerKey.
  • The e2e tests that need no credentials: 47 pass, 4 skipped (they need credentials). New, with no network and no credentials:
    • test/e2e/s3-local-flag.test.js: each of the six commands with --local in S3 mode stops before it connects. The endpoint is a port where nothing listens, so a command that connects fails with ECONNREFUSED. Without the fix, all six fail.
    • test/e2e/same-content.test.js: a --local publish of 40 files with the same content and 1 other file. It expects 2 Brotli and 2 gzip compressions, and 6 variants to upload. Without the fix, there are more compressions.
  • test/e2e/kv-store.test.js and test/e2e/s3.test.js with real test resources: 24 pass. deployed.test.js was not run.
    • The shared fixture now has 3 copies of style.css. After the first publish, these tests compare the number of variants to upload with the number of file keys in storage. Before the fix, both tests fail (23 uploads for 17 keys).
  • By hand: in S3 mode, collections delete --local with a Service ID stops with the --local error before it connects to storage.

🤖 Generated with Claude Code

@harmony7 harmony7 changed the title Fix S3 mode issues for 8.0: purge after delete, --local, and duplicate variants 8.0 fixes: purge after collections delete, --local in S3 mode, and duplicate variants Oct 20, 2026
@harmony7
harmony7 requested review from a team and kailan and removed request for a team October 20, 2026 12:16
harmony7 and others added 3 commits October 22, 2026 09:15
After a collection is deleted, the service can continue to serve its cached responses until they expire. collections delete now purges the surrogate key <publishId>-<collection> after the delete, with the same options and rules as publish-content: --fastly-service-id, --purge-environment, and the API token check before anything is deleted.

The purge loop moves from publish-content to purgeCollection() in util/purge.ts, so that both commands use it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In S3 mode, the commands ignored --local for storage. publish-content
uploaded to the bucket, and collections delete and clean deleted from
it. There is no local simulation of S3 storage.

The S3 storage provider builder now throws an error if --local is set,
before it prints or connects to anything. All commands that accept
--local (publish-content, collections list, delete, promote,
update-expiration, and clean) load the provider through this builder,
so they all stop with exit code 1.

--help and the README now tell that --local is for KV Store mode only.

The new e2e test needs no network and no credentials. It uses an
endpoint where nothing listens, so a command that connects fails with
ECONNREFUSED.

Fixes #111.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
publish-content scans up to 16 files at the same time. Files with the
same content checked the variant map before the first of them set it,
because the map was set only after the compression and the hash. Thus
each of these files compressed the content again (mostly visible for
Brotli), and added the same keys to the upload batch. Several files
could also write the same variant file at the same time, and the hash
could read a file that another write had truncated.

The new OncePerKey helper keeps one promise for each variant storage
key. The first file starts the work and stores the promise before its
first await. The other files await the same promise. Only the first
file adds the key to the batch, and the variant file is written one
time.

publish-content --local now shows the number of variants to upload,
because it does not list the keys in storage and showed no count
before.

Tests: a unit test for OncePerKey, and an e2e test (no network, no
credentials) that publishes 40 files with the same content and 1 other
file with --local. It expects 2 Brotli and 2 gzip compressions and 6
variants to upload. Before the fix, it had many more compressions.

Fixes #112.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant