diff --git a/CHANGELOG.md b/CHANGELOG.md index d97e6ce..a601ea2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -27,6 +27,9 @@ This file does not list v7 releases after 7.0.8, or v6 releases after 6.3.0. For - Add `--purge-environment` to purge `production` (the default), `staging`, or both. A staging purge sends the `Fastly-Purge-Environment: staging` header. - A failed purge shows a warning on stderr. The command still completes, because the content is already published. +- `collections delete` + - After the delete, purge the surrogate key of the collection, so that the service stops serving cached responses of the deleted collection. This has the same options and rules as the purge of `publish-content`: `--fastly-service-id`, `--purge-environment`, and the API token. + - S3-compatible storage (BETA) - Add support for S3-compatible storage, such as Fastly Object Storage - Store items using same keys as KV Store @@ -107,6 +110,11 @@ This file does not list v7 releases after 7.0.8, or v6 releases after 6.3.0. For - `collectionSelector.fromCookie()`: the activate and reset paths now accept only a path on the same site for `redirectTo`. Any other value redirects to `/`. The activate path returns `400` for a `collection` value with characters that a cookie value cannot have, and sets no cookie. +- S3-compatible storage: `--local` stops the command with an error before it connects to storage. Before, `publish-content`, `collections list`, `collections delete`, `collections promote`, `collections update-expiration`, and `clean` ignored `--local` for storage, and changed the bucket. There is no local simulation of S3 storage. If you used `--local` in S3 mode to skip the purge after `publish-content`, remove `--local`, and do not set a Service ID. `--help` and the README now tell that `--local` is for KV Store mode only. + +- `publish-content` prepares each variant of the same content one time. Before, when several files had the same content, the files that were processed at the same time each compressed the content again, and each added the same keys to the upload. The uploaded content was correct, but the publish used more CPU time and more upload requests (in S3 mode, one `PutObject` for each duplicate). Each variant file in the working directory is now also written one time, so a file is not read while another file writes it. + - `publish-content --local` shows the number of variants to upload. + ## [7.0.8] - 2026-10-14 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index 4fa199d..a91b591 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,7 +43,7 @@ Both parts use a registry of builder functions. Each entry point calls `register - `kv-store-provider` uses the Fastly API. - `kv-store-local-provider` is for the `--local` flag. It writes `static-publisher/kvstore.json` and the prepared files, so that `fastly compute serve` can simulate the KV Store. - `s3-storage-provider` uses the AWS SDK. - - The CLI interface has list, get, submit, delete, batch, and chunking. The purge after publishing is not part of it: `publish-content` does it with `util/purge.ts`, because it is the same for every storage mode. + - The CLI interface has list, get, submit, delete, batch, and chunking. The purge after a change to a collection is not part of it: `publish-content` and `collections delete` do it with `util/purge.ts`, because it is the same for every storage mode. - The Fastly API token comes only from `--fastly-api-token` or `FASTLY_API_TOKEN` (`util/api-token.ts`). As of v8, the CLI does not run `fastly profile token`, and the package does not depend on `@fastly/cli`. (The scaffolder still adds `@fastly/cli` to the generated app.) - Server (`src/server/storage/`): `kv-store-provider` and `s3-storage-provider`. Both use a small `getEntry(key, tags)` interface. S3 credentials come from a Secret Store. `src/server/index.ts` exports the setters. - The server's `s3-storage-provider` does not use the AWS SDK. It signs a `GET` with `@smithy/signature-v4` and reads the status code, the body, and the `x-amz-meta-*` headers. The SDK's browser build, which js-compute bundles, parses XML with `DOMParser` (since `@aws-sdk/xml-builder` 3.894.0), and Compute does not have `DOMParser`. Do not import `@aws-sdk/*` in `src/server`. @@ -56,7 +56,7 @@ All keys start with `publishId` (default `'default'`): - `_settings_`: The normalized `server` section of `publish-content.config.js`. The CLI saves it when it publishes. - `_files_sha256_[_]`: The file content, with `br`/`gzip` variants. The key is the hash of the content. Large files are split into chunks. -All collections share the files, because the key is the hash of the content. `clean` removes expired collections and the `_files_` keys that no index uses. After a publish, the CLI purges the surrogate key `-`. The server adds this tag when it reads, and to the responses in the response cache. The purge runs in KV Store and S3 modes (not with `--local`), only if the CLI finds a Service ID (`--fastly-service-id`, then `FASTLY_SERVICE_ID`, then `service_id` in `fastly.toml`, the same order as the Fastly CLI; see `util/service-id.ts`). `loadPurgeTarget()` checks this before the upload, and fails if there is a Service ID but no API token. It is a soft purge. A purge without the `Fastly-Purge-Environment: staging` header does not clear the cache of a staged service version: `--purge-environment` selects `production` (the default, no header), `staging`, or both, and `publish-content` sends one purge for each. A failed purge is only a warning, because the content is already published. +All collections share the files, because the key is the hash of the content. `clean` removes expired collections and the `_files_` keys that no index uses. After a publish or a `collections delete`, the CLI purges the surrogate key `-` (`purgeCollection()`). The server adds this tag when it reads, and to the responses in the response cache. The purge runs in KV Store and S3 modes (not with `--local`), only if the CLI finds a Service ID (`--fastly-service-id`, then `FASTLY_SERVICE_ID`, then `service_id` in `fastly.toml`, the same order as the Fastly CLI; see `util/service-id.ts`). `loadPurgeTarget()` checks this before the upload or the delete, and fails if there is a Service ID but no API token. It is a soft purge. A purge without the `Fastly-Purge-Environment: staging` header does not clear the cache of a staged service version: `--purge-environment` selects `production` (the default, no header), `staging`, or both, and the CLI sends one purge for each. A failed purge is only a warning, because the change to storage is already done. ### `publish-content` flow diff --git a/MIGRATING.md b/MIGRATING.md index 19f407b..a86415a 100644 --- a/MIGRATING.md +++ b/MIGRATING.md @@ -38,7 +38,8 @@ To upgrade an app that uses the KV Store: in KV Store mode too. It finds the Service ID from `--fastly-service-id`, then the `FASTLY_SERVICE_ID` environment variable, then `service_id` in `fastly.toml`. If it finds a Service ID, the API token must have permission to purge that service. If you do not want a purge, make sure that the CLI does not find a Service ID. If the purge fails, - the CLI shows a warning, and the publish is complete. + the CLI shows a warning, and the publish is complete. `collections delete` purges the same way after it deletes a + collection. - **`kvStoreAssetInclusionTest` has a new name.** In `publish-content.config.js`, use `assetInclusionTest`. The old name continues to work, but it is deprecated. - **Some types have new names.** `KVAssetEntry` is now `AssetEntry`, and `KVAssetEntryMap` is now `AssetEntryMap`. diff --git a/README.md b/README.md index 37321de..8dbb733 100644 --- a/README.md +++ b/README.md @@ -935,7 +935,7 @@ After this process is complete, the PublisherServer object in the Compute applic **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. @@ -962,7 +962,7 @@ This can include expired collection indexes and orphaned content assets. **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. @@ -981,7 +981,7 @@ Lists all collections currently published in storage. **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. @@ -1014,7 +1014,7 @@ Copies an existing collection (content + config) to a new collection name. **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. @@ -1044,7 +1044,7 @@ Sets or updates the expiration time of an existing collection. **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. @@ -1063,13 +1063,19 @@ Deletes a collection index from storage. The content files will remain as they m Use the `npx @fastly/compute-js-static-publish clean` command afterward to remove content files that are no longer referenced by any collection. +If a Service ID is found, the command then purges the surrogate key `-`, so that the service stops serving cached responses of the deleted collection. This uses the same options as [`publish-content`](#publish-content). + ##### Options: - `--collection-name`: The name of the collection to delete (required) +- `--fastly-service-id`: The Fastly Service ID to purge after deleting. If not set, the tool checks the `FASTLY_SERVICE_ID` environment variable, then `service_id` in `fastly.toml`. If none is found, the command skips the purge. The purge also needs an API token (see `--fastly-api-token`). With `--local`, there is no purge. If the purge fails, the command shows a warning and completes, because the collection is already deleted. + +- `--purge-environment`: The environments to purge: `production` (the default), `staging`, or both, as a comma-separated list. You can also repeat the option. + **KV Store Options:** -- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. +- `--local`: Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. KV Store mode only. In S3 mode, `--local` is an error, because there is no local simulation of S3 storage. - `--fastly-api-token`: API token to use when publishing. If not set, the tool uses the **`FASTLY_API_TOKEN` environment variable**. diff --git a/src/cli/commands/manage/clean.ts b/src/cli/commands/manage/clean.ts index 508397f..9cc0a73 100644 --- a/src/cli/commands/manage/clean.ts +++ b/src/cli/commands/manage/clean.ts @@ -33,6 +33,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/collections/delete.ts b/src/cli/commands/manage/collections/delete.ts index 85a282a..739462f 100644 --- a/src/cli/commands/manage/collections/delete.ts +++ b/src/cli/commands/manage/collections/delete.ts @@ -10,6 +10,7 @@ import { type OptionDefinition } from 'command-line-args'; import { parseCommandLine } from '../../../util/args.js'; import { LoadConfigError, loadStaticPublisherRcFile } from '../../../util/config.js'; import { loadStorageProviderFromStaticPublishRc } from '../../../storage/storage-provider.js'; +import { type PurgeTarget, loadPurgeTarget, purgeCollection } from '../../../util/purge.js'; function help() { console.log(`\ @@ -24,6 +25,9 @@ Description: Use the 'npx @fastly/compute-js-static-publish clean' command afterward to remove content files that are no longer referenced by any collection. + If a Service ID is found, the tool then purges the cached responses of the collection + from that service. + Required: --collection-name= The name of the collection to delete @@ -31,11 +35,23 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. - --fastly-api-token= Fastly API token for KV Store access. + --fastly-api-token= Fastly API token for KV Store or cache access. If not set, the tool uses the FASTLY_API_TOKEN environment variable. +Purge Options: + --fastly-service-id= Fastly Service ID to purge after deleting (not with --local). + If not set, the tool will check: + 1. FASTLY_SERVICE_ID environment variable + 2. service_id in fastly.toml + If none is found, the purge is skipped. + + --purge-environment= Environment to purge: production, staging, or a + comma-separated list of both. Can be repeated. + Default: production + S3 Storage Options (BETA): --s3-access-key-id= Access Key ID and Secret Access Key used to --s3-secret-access-key= interface with S3 or compatible storage. @@ -56,6 +72,8 @@ export async function action(actionArgs: string[]) { { name: 'local', type: Boolean }, { name: 'fastly-api-token', type: String, }, + { name: 'fastly-service-id', type: String, }, + { name: 'purge-environment', type: String, multiple: true, }, { name: 's3-access-key-id', type: String, }, { name: 's3-secret-access-key', type: String, }, @@ -77,6 +95,8 @@ export async function action(actionArgs: string[]) { verbose, ['collection-name']: collectionNameValue, ['fastly-api-token']: fastlyApiToken, + ['fastly-service-id']: fastlyServiceId, + ['purge-environment']: purgeEnvironmentValues, local: localMode, ['s3-access-key-id']: s3AccessKeyId, ['s3-secret-access-key']: s3SecretAccessKey, @@ -145,6 +165,22 @@ export async function action(actionArgs: string[]) { console.log(`✔️ Collection to delete: ${collectionName}`); + // Purge target: checked now, so that a missing API token fails before anything is deleted. + let purgeTarget: PurgeTarget | null; + try { + purgeTarget = loadPurgeTarget({ + localMode: localMode ?? false, + computeAppDir, + fastlyServiceId, + fastlyApiToken, + purgeEnvironments: purgeEnvironmentValues, + }); + } catch (err: unknown) { + console.error(String(err)); + process.exitCode = 1; + return; + } + // ### Asset Keys to delete const assetKeysToDelete = new Set(); @@ -181,5 +217,9 @@ export async function action(actionArgs: string[]) { true, ) + if (purgeTarget != null) { + await purgeCollection(purgeTarget, publishId, collectionName); + } + console.log("✅ Completed.") } diff --git a/src/cli/commands/manage/collections/index.ts b/src/cli/commands/manage/collections/index.ts index cef2002..82ec8a7 100644 --- a/src/cli/commands/manage/collections/index.ts +++ b/src/cli/commands/manage/collections/index.ts @@ -29,6 +29,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/collections/list.ts b/src/cli/commands/manage/collections/list.ts index fc7fc7d..be05d5c 100644 --- a/src/cli/commands/manage/collections/list.ts +++ b/src/cli/commands/manage/collections/list.ts @@ -26,6 +26,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/collections/promote.ts b/src/cli/commands/manage/collections/promote.ts index 14f4d69..8a14144 100644 --- a/src/cli/commands/manage/collections/promote.ts +++ b/src/cli/commands/manage/collections/promote.ts @@ -48,6 +48,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/collections/update-expiration.ts b/src/cli/commands/manage/collections/update-expiration.ts index 7ee449e..b29793e 100644 --- a/src/cli/commands/manage/collections/update-expiration.ts +++ b/src/cli/commands/manage/collections/update-expiration.ts @@ -44,6 +44,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/index.ts b/src/cli/commands/manage/index.ts index 1a798a4..d6e21cf 100644 --- a/src/cli/commands/manage/index.ts +++ b/src/cli/commands/manage/index.ts @@ -31,6 +31,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --fastly-api-token= Fastly API token for KV Store access. If not set, the tool uses the FASTLY_API_TOKEN diff --git a/src/cli/commands/manage/publish-content.ts b/src/cli/commands/manage/publish-content.ts index 6a12e23..0404972 100644 --- a/src/cli/commands/manage/publish-content.ts +++ b/src/cli/commands/manage/publish-content.ts @@ -11,7 +11,7 @@ import { type OptionDefinition } from 'command-line-args'; import { type AssetEntryMap, } from '../../../models/assets/index.js'; import { type ContentCompressionTypes } from '../../../models/compression/index.js'; import { type PublisherServerConfigNormalized } from '../../../models/config/publisher-server-config.js'; -import { type ContentTypeDef } from '../../../models/config/publish-content-config.js'; +import { type ContentTypeDef, type PublishContentConfigNormalized } from '../../../models/config/publish-content-config.js'; import { encodeIndexMetadata, type IndexMetadata } from '../../../models/server/index.js'; import { calcExpirationTime } from '../../../models/time/index.js'; import { parseCommandLine } from '../../util/args.js'; @@ -19,9 +19,9 @@ import { mergeContentTypes, testFileContentType } from '../../util/content-types import { LoadConfigError, loadPublishContentConfigFile, loadStaticPublisherRcFile } from '../../util/config.js'; import { applyDefaults } from '../../util/data.js'; import { calculateFileSizeAndHash, enumerateFiles, rootRelative } from '../../util/files.js'; -import { ensureVariantFileExists, type Variants } from '../../util/variants.js'; +import { ensureVariantFileExists, OncePerKey, type Variants } from '../../util/variants.js'; import { concurrentMap } from '../../util/retryable.js'; -import { type PurgeTarget, loadPurgeTarget, purgeSurrogateKey } from '../../util/purge.js'; +import { type PurgeTarget, loadPurgeTarget, purgeCollection } from '../../util/purge.js'; import { getStorageKeysByHexPrefix, loadStorageProviderFromStaticPublishRc, @@ -96,6 +96,7 @@ KV Store Options: --local Instead of working with the Fastly KV Store, operate on local files that will be used to simulate the KV Store with the local development environment. + KV Store mode only. In S3 mode, --local is an error. --kv-overwrite Alias for --overwrite-existing. @@ -214,7 +215,7 @@ export async function action(actionArgs: string[]) { const configFilePath = configFilePathValue ?? './publish-content.config.js'; - let publishContentConfig; + let publishContentConfig: PublishContentConfigNormalized; try { publishContentConfig = await loadPublishContentConfigFile(configFilePath); } catch (err) { @@ -372,9 +373,10 @@ export async function action(actionArgs: string[]) { const assetsIndex: AssetEntryMap = {}; // For each variant that we know about during this publishing, true if the - // index keeps the variant. Keyed on the base version's hash. - type VariantKeepMap = Map; - const baseHashToVariantKeepMap = new Map(); + // index keeps the variant. Keyed on the variant's storage key. Files with the + // same content have the same keys, so the first of them prepares each variant + // one time, and the other files await its result. + const variantKeeps = new OncePerKey(); // #### Keys that are already in storage // The key of a file is the hash of its content. If the key exists, storage @@ -448,13 +450,6 @@ export async function action(actionArgs: string[]) { const stats = fs.statSync(file); const lastModifiedTime = Math.floor((stats.mtime).getTime() / 1000); - // #### Keep flag per variant - let variantKeeps = baseHashToVariantKeepMap.get(baseHash); - if (variantKeeps == null) { - variantKeeps = new Map(); - baseHashToVariantKeepMap.set(baseHash, variantKeeps); - } - // We know the number of chunks of the original before we read it. For a // compressed variant, we know it only if the original fits in one chunk. const baseNumChunks = storageProvider.calculateNumChunks(baseSize); @@ -468,24 +463,10 @@ export async function action(actionArgs: string[]) { const batchItems: StorageProviderBatchEntry[] = []; - for (const variant of variants) { - let variantKey = `${publishId}_files_sha256_${baseHash}`; - let variantFilename = `${baseHash}`; - if (variant !== 'original') { - variantKey = `${variantKey}_${variant}`; - variantFilename = `${variantFilename}_${variant}`; - } - - const variantFilePath = path.resolve(storageContentDir, variantFilename); - - let keep = variantKeeps.get(variant); - if (keep != null) { - - if (verbose) { - console.log(` 🏃‍♂️ Asset "${variantKey}" is identical to an item we already know about, reusing existing copy.`); - } - - } else if ( + // Finds the variant in storage, or prepares it and adds it to batchItems. + // Returns true if the index keeps the variant. + async function prepareVariant(variant: Variants, variantKey: string, variantFilePath: string) { + if ( (variant === 'original' || baseNumChunks === 1) && isVariantInStorage(variantKey, variant === 'original' ? baseNumChunks : 1) ) { @@ -496,58 +477,76 @@ export async function action(actionArgs: string[]) { existingVariantCount++; // We upload a compressed variant only if it is smaller than the // original. Thus if it is in storage, the index keeps it. - keep = true; - variantKeeps.set(variant, keep); + return true; + + } + await ensureVariantFileExists( + variantFilePath, + variant, + file, + verbose, + { brotliQuality: publishContentConfig.brotliQuality }, + ); + + let contentEncoding, hash, size; + if (variant === 'original') { + contentEncoding = undefined; + hash = baseHash; + size = baseSize; } else { + contentEncoding = variant; + ({hash, size} = await calculateFileSizeAndHash(variantFilePath)); + } - await ensureVariantFileExists( - variantFilePath, - variant, - file, - verbose, - { brotliQuality: publishContentConfig.brotliQuality }, - ); - - let contentEncoding, hash, size; - if (variant === 'original') { - contentEncoding = undefined; - hash = baseHash; - size = baseSize; - } else { - contentEncoding = variant; - ({hash, size} = await calculateFileSizeAndHash(variantFilePath)); - } + // Only keep variants whose file size actually ends up smaller than + // what we started with. Do not upload the other variants. + const keep = variant === 'original' || size < baseSize; - // Only keep variants whose file size actually ends up smaller than - // what we started with. Do not upload the other variants. - keep = variant === 'original' || size < baseSize; - variantKeeps.set(variant, keep); - - if (keep) { - const numChunks = storageProvider.calculateNumChunks(size); - - const metadataJson: Record = { - size: String(size), - hash, - }; - if (contentEncoding != null) { - metadataJson.contentEncoding = contentEncoding; - } - if (numChunks > 1) { - metadataJson.numChunks = String(numChunks); - } - - batchItems.push({ - size, - key: variantKey, - filePath: variantFilePath, - metadataJson, - }); - } else if (verbose) { - console.log(` ・ Variant "${variantKey}" is not smaller than the original, not uploading.`); + if (keep) { + const numChunks = storageProvider.calculateNumChunks(size); + + const metadataJson: Record = { + size: String(size), + hash, + }; + if (contentEncoding != null) { + metadataJson.contentEncoding = contentEncoding; + } + if (numChunks > 1) { + metadataJson.numChunks = String(numChunks); } + + batchItems.push({ + size, + key: variantKey, + filePath: variantFilePath, + metadataJson, + }); + } else if (verbose) { + console.log(` ・ Variant "${variantKey}" is not smaller than the original, not uploading.`); + } + return keep; + } + + for (const variant of variants) { + let variantKey = `${publishId}_files_sha256_${baseHash}`; + let variantFilename = `${baseHash}`; + if (variant !== 'original') { + variantKey = `${variantKey}_${variant}`; + variantFilename = `${variantFilename}_${variant}`; + } + + const variantFilePath = path.resolve(storageContentDir, variantFilename); + + const { promise, first } = variantKeeps.run( + variantKey, + () => prepareVariant(variant, variantKey, variantFilePath), + ); + if (!first && verbose) { + console.log(` 🏃‍♂️ Asset "${variantKey}" is identical to an item we already know about, reusing existing copy.`); } + const keep = await promise; if (variant !== 'original' && keep) { variantsToKeep.push(variant); @@ -579,6 +578,8 @@ export async function action(actionArgs: string[]) { console.log(`✅ Scan complete.`); if (existingKeys != null) { console.log(` | ${existingVariantCount} variant(s) already in storage, ${batch.storageProviderBatchEntries.length} to upload.`); + } else { + console.log(` | ${batch.storageProviderBatchEntries.length} variant(s) to upload.`); } await storageProvider.applyBatch(batch, { existingKeys }); @@ -685,17 +686,7 @@ export async function action(actionArgs: string[]) { console.log(`✅ Settings have been saved.`); if (purgeTarget != null) { - const surrogateKey = `${publishId}-${collectionName}`; - for (const environment of purgeTarget.environments) { - console.log(`Purging surrogate key [${surrogateKey}] on service [${purgeTarget.serviceId}] (${environment})...`); - const purged = await purgeSurrogateKey(purgeTarget.fastlyApiContext, purgeTarget.serviceId, surrogateKey, true, environment); - if (purged) { - console.log('Purged'); - } else { - // The content is published. Only the cached copies may be stale until they expire. - console.warn(`⚠️ Warning: Failed purging (${environment}). Cached copies may be served until they expire.`); - } - } + await purgeCollection(purgeTarget, publishId, collectionName); } console.log(`🎉 Completed.`); diff --git a/src/cli/storage/s3-storage-provider.ts b/src/cli/storage/s3-storage-provider.ts index 65fe221..1468d31 100644 --- a/src/cli/storage/s3-storage-provider.ts +++ b/src/cli/storage/s3-storage-provider.ts @@ -60,15 +60,21 @@ type CommandOutput = C extends Command ? O : nev // objects, 64 was approximately 4 times faster than 12. export const DEFAULT_S3_UPLOAD_CONCURRENCY = 64; +export const S3_LOCAL_MODE_ERROR = '--local is available only in KV Store mode. In S3 mode, the commands always use the bucket in static-publish.rc.js.'; + export const buildStoreProvider: StorageProviderBuilder = async ( config: StaticPublishRc, context: StorageProviderBuilderContext, ) => { - if (isS3StorageConfigRc(config)) { - console.log(` Working on S3 (or compatible) storage (BETA)...`); - } else { + if (!isS3StorageConfigRc(config)) { return null; } + // There is no local simulation of S3 storage. Stop before the command + // connects to the bucket. + if (context.localMode) { + throw new Error(S3_LOCAL_MODE_ERROR); + } + console.log(` Working on S3 (or compatible) storage (BETA)...`); const { region, diff --git a/src/cli/util/purge.ts b/src/cli/util/purge.ts index b33d196..7554790 100644 --- a/src/cli/util/purge.ts +++ b/src/cli/util/purge.ts @@ -47,17 +47,17 @@ export type LoadPurgeTargetParams = { purgeEnvironments: unknown, }; -// Finds the service to purge after publishing, so that it stops serving cached -// copies of the collection. Returns null, and logs why, if the purge is skipped. -// Throws if a Service ID is found but an API token is not, so that this fails -// before anything is uploaded. +// Finds the service to purge after a change to a collection, so that it stops +// serving cached copies of the collection. Returns null, and logs why, if the +// purge is skipped. Throws if a Service ID is found but an API token is not, so +// that a command can fail before it changes storage. export function loadPurgeTarget(params: LoadPurgeTargetParams): PurgeTarget | null { // Check the option even when the purge is skipped, so that a typo is reported. const environments = parsePurgeEnvironments(params.purgeEnvironments); if (params.localMode) { - console.log(`- Local mode: will skip purge step after publish.`); + console.log(`- Local mode: will skip the purge.`); return null; } @@ -66,7 +66,7 @@ export function loadPurgeTarget(params: LoadPurgeTargetParams): PurgeTarget | nu fastlyTomlPath: path.resolve(params.computeAppDir, 'fastly.toml'), }); if (serviceIdResult == null) { - console.log(`- Service ID not found (--fastly-service-id, FASTLY_SERVICE_ID, or fastly.toml). Will skip purge step after publish.`); + console.log(`- Service ID not found (--fastly-service-id, FASTLY_SERVICE_ID, or fastly.toml). Will skip the purge.`); return null; } console.log(`✔️ Service ID from ${serviceIdResult.source}: ${serviceIdResult.serviceId}`); @@ -118,4 +118,19 @@ export async function purgeSurrogateKey( return true; -} \ No newline at end of file +} + +// Purges the cached responses of a collection in each environment of the purge +// target. A failed purge is a warning, because the change to storage is done. +export async function purgeCollection(purgeTarget: PurgeTarget, publishId: string, collectionName: string) { + const surrogateKey = `${publishId}-${collectionName}`; + for (const environment of purgeTarget.environments) { + console.log(`Purging surrogate key [${surrogateKey}] on service [${purgeTarget.serviceId}] (${environment})...`); + const purged = await purgeSurrogateKey(purgeTarget.fastlyApiContext, purgeTarget.serviceId, surrogateKey, true, environment); + if (purged) { + console.log('Purged'); + } else { + console.warn(`⚠️ Warning: Failed purging (${environment}). Cached copies may be served until they expire.`); + } + } +} diff --git a/src/cli/util/service-id.ts b/src/cli/util/service-id.ts index a0370ad..c2188fe 100644 --- a/src/cli/util/service-id.ts +++ b/src/cli/util/service-id.ts @@ -17,7 +17,7 @@ export type LoadServiceIdParams = { fastlyTomlPath: string, }; -// Finds the Fastly Service ID to purge after publishing. Checks, in order: +// Finds the Fastly Service ID to purge after a change to a collection. Checks, in order: // 1. The command line (--fastly-service-id) // 2. The FASTLY_SERVICE_ID environment variable // 3. service_id in fastly.toml diff --git a/src/cli/util/variants.ts b/src/cli/util/variants.ts index 38e70c2..4106c0a 100644 --- a/src/cli/util/variants.ts +++ b/src/cli/util/variants.ts @@ -10,6 +10,23 @@ import { rootRelative } from './files.js'; export type Variants = 'original' | ContentCompressionTypes; +// Runs an async function one time for each key. The first call for a key +// starts the function and stores its promise before it returns. Later calls +// for the same key, also concurrent calls, get the same promise. +export class OncePerKey { + private readonly promises = new Map>(); + + run(key: string, fn: () => Promise): { promise: Promise, first: boolean } { + let promise = this.promises.get(key); + if (promise != null) { + return { promise, first: false }; + } + promise = fn(); + this.promises.set(key, promise); + return { promise, first: true }; + } +} + export async function ensureVariantFileExists( variantFilePath: string, variant: Variants, diff --git a/test/e2e/helpers.js b/test/e2e/helpers.js index c64a97c..cbda9e8 100644 --- a/test/e2e/helpers.js +++ b/test/e2e/helpers.js @@ -56,7 +56,7 @@ export function removeDir(dir) { } // Content to publish. The files test content types, auto index, auto -// extensions, compression, and the 404 page. +// extensions, compression, the 404 page, and files with the same content. export const LARGE_TEXT = 'The quick brown fox jumps over the lazy dog.\n'.repeat(2000); export function writeFixtureContent(publicDir, { largeBinaryBytes = 0 } = {}) { const files = { @@ -66,6 +66,9 @@ export function writeFixtureContent(publicDir, { largeBinaryBytes = 0 } = {}) { 'cached.html': 'Cached

Cached

', '404.html': 'Not found

Not found

', 'style.css': LARGE_TEXT, + 'copies/style-1.css': LARGE_TEXT, + 'copies/style-2.css': LARGE_TEXT, + 'copies/style-3.css': LARGE_TEXT, 'image.tiff': crypto.randomBytes(256), 'image.avif': crypto.randomBytes(256), 'module.wasm': crypto.randomBytes(256), diff --git a/test/e2e/s3-local-flag.test.js b/test/e2e/s3-local-flag.test.js new file mode 100644 index 0000000..554f9c9 --- /dev/null +++ b/test/e2e/s3-local-flag.test.js @@ -0,0 +1,78 @@ +/* + * Copyright Fastly, Inc. + * Licensed under the MIT license. See LICENSE file for details. + */ + +// In S3 mode, each command that accepts --local stops with an error before it +// connects to the bucket. The endpoint is a port where nothing listens, so a +// command that connects fails with ECONNREFUSED. No network and no credentials +// are necessary. + +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import { after, before, describe, test } from 'node:test'; + +import { makeTempDir, removeDir, runCli, writeFixtureContent } from './helpers.js'; + +const COMMANDS = [ + [ 'publish-content', '--overwrite-existing' ], + [ 'collections', 'list' ], + [ 'collections', 'delete', '--collection-name=x' ], + [ 'collections', 'promote', '--collection-name=x', '--to=y' ], + [ 'collections', 'update-expiration', '--collection-name=x', '--expires-never' ], + [ 'clean' ], +]; + +describe('--local in S3 mode', () => { + let dir; + + before(() => { + dir = makeTempDir('s3-local-flag'); + writeFixtureContent(path.resolve(dir, 'public')); + fs.writeFileSync(path.resolve(dir, 'static-publish.rc.js'), `\ +export default { + storageMode: 's3', + s3: { region: 'us-east-1', bucket: 'test', endpoint: 'http://127.0.0.1:9' }, + publishId: 'default', + defaultCollectionName: 'live', + staticPublisherWorkingDir: './static-publisher', +}; +`); + fs.writeFileSync(path.resolve(dir, 'publish-content.config.js'), `\ +export default { + rootDir: './public', +}; +`); + }); + + after(() => { + if (dir != null) { + removeDir(dir); + } + }); + + for (const args of COMMANDS) { + test(`${args.slice(0, 2).join(' ')}: stops before it connects`, () => { + const result = runCli([ ...args, '--local' ], { + cwd: dir, + env: { S3_ACCESS_KEY_ID: 'dummy', S3_SECRET_ACCESS_KEY: 'dummy' }, + allowFailure: true, + }); + assert.equal(result.status, 1, result.output); + assert.match(result.output, /--local is available only in KV Store mode/); + assert.doesNotMatch(result.output, /ECONNREFUSED|Working on S3/); + }); + } + + test('without --local: the command connects to the bucket', () => { + const result = runCli([ 'collections', 'list' ], { + cwd: dir, + env: { S3_ACCESS_KEY_ID: 'dummy', S3_SECRET_ACCESS_KEY: 'dummy' }, + allowFailure: true, + }); + assert.notEqual(result.status, 0); + assert.match(result.output, /Working on S3/); + assert.doesNotMatch(result.output, /--local is available only/); + }); +}); diff --git a/test/e2e/same-content.test.js b/test/e2e/same-content.test.js new file mode 100644 index 0000000..85440e0 --- /dev/null +++ b/test/e2e/same-content.test.js @@ -0,0 +1,83 @@ +/* + * Copyright Fastly, Inc. + * Licensed under the MIT license. See LICENSE file for details. + */ + +// publish-content with many files that have the same content: each variant is +// prepared one time, and each key is in the batch one time. Uses --local with +// a hand-written static-publish.rc.js. No network and no credentials are +// necessary. + +import assert from 'node:assert/strict'; +import crypto from 'node:crypto'; +import fs from 'node:fs'; +import path from 'node:path'; +import { after, before, describe, test } from 'node:test'; + +import { makeTempDir, removeDir, runCli } from './helpers.js'; + +const SAME_FILE_COUNT = 40; + +function countMatches(output, regex) { + return output.split('\n').filter(line => regex.test(line)).length; +} + +describe('publish-content: files with the same content', () => { + let dir; + let output; + + before(() => { + dir = makeTempDir('same-content'); + const publicDir = path.resolve(dir, 'public'); + fs.mkdirSync(publicDir, { recursive: true }); + // Text that compresses, but not too fast, so that the files overlap in time. + const words = Array.from({ length: 20000 }, () => crypto.randomBytes(4).toString('hex')); + const sameContent = words.join(' '); + for (let i = 0; i < SAME_FILE_COUNT; i++) { + fs.writeFileSync(path.resolve(publicDir, `page-${i}.html`), sameContent); + } + fs.writeFileSync(path.resolve(publicDir, 'other.html'), sameContent + ' other'); + + fs.writeFileSync(path.resolve(dir, 'static-publish.rc.js'), `\ +export default { + kvStoreName: 'same-content', + publishId: 'default', + defaultCollectionName: 'live', + staticPublisherWorkingDir: './static-publisher', +}; +`); + fs.writeFileSync(path.resolve(dir, 'publish-content.config.js'), `\ +export default { + rootDir: './public', +}; +`); + ({ output } = runCli([ 'publish-content', '--local', '--verbose' ], { cwd: dir })); + }); + + after(() => { + if (dir != null) { + removeDir(dir); + } + }); + + test('prepares each variant one time', () => { + assert.equal(countMatches(output, /Copied file/), 2, output); + assert.equal(countMatches(output, /Compressed file .*\[br\]/), 2, output); + assert.equal(countMatches(output, /Compressed file .*\[gzip\]/), 2, output); + }); + + test('has each key one time in the batch', () => { + // 2 originals, 2 br, and 2 gzip. + assert.match(output, /\| 6 variant\(s\) to upload\./); + }); + + test('indexes all files', () => { + const index = JSON.parse(fs.readFileSync(path.resolve(dir, 'static-publisher/storage-content/index_live.json'), 'utf-8')); + assert.equal(Object.keys(index).length, SAME_FILE_COUNT + 1); + const keys = new Set(Object.values(index).map(entry => entry.key)); + assert.equal(keys.size, 2); + for (const entry of Object.values(index)) { + assert.deepEqual([ ...entry.variants ].sort(), [ 'br', 'gzip' ]); + } + }); +}); diff --git a/test/e2e/storage-publish.js b/test/e2e/storage-publish.js index fe63991..d9aa89f 100644 --- a/test/e2e/storage-publish.js +++ b/test/e2e/storage-publish.js @@ -137,6 +137,10 @@ export function defineStoragePublishTests({ storageMode, scaffoldArgs, serve = f assert.equal(summary.existing, 0); assert.ok(summary.toUpload > 0); const keys = await listKeys(); + // Files with the same content have the same keys. Each key is uploaded one + // time. A chunk of a large file (`_1`, `_2`, ...) is not an upload entry. + const fileKeys = keys.filter(key => key.startsWith(`${publishId}_files_sha256_`) && !/_\d+$/.test(key)); + assert.equal(summary.toUpload, fileKeys.length, 'each key is uploaded one time'); assert.ok(keys.includes(`${publishId}_index_${COLLECTION_NAME}`), 'has the index'); assert.ok(keys.includes(`${publishId}_settings_${COLLECTION_NAME}`), 'has the settings'); }); diff --git a/test/unit/once-per-key.test.js b/test/unit/once-per-key.test.js new file mode 100644 index 0000000..53b08ef --- /dev/null +++ b/test/unit/once-per-key.test.js @@ -0,0 +1,52 @@ +/* + * Copyright Fastly, Inc. + * Licensed under the MIT license. See LICENSE file for details. + */ + +import assert from 'node:assert/strict'; +import { test } from 'node:test'; + +import { OncePerKey } from '../../build/cli/cli/util/variants.js'; + +test('concurrent calls with the same key run the function one time', async () => { + const once = new OncePerKey(); + let calls = 0; + let release; + const gate = new Promise(resolve => { release = resolve; }); + const fn = async () => { + calls++; + await gate; + return 'result'; + }; + + const results = [ once.run('a', fn), once.run('a', fn), once.run('a', fn) ]; + assert.deepEqual(results.map(r => r.first), [ true, false, false ]); + assert.equal(calls, 1); + release(); + assert.deepEqual(await Promise.all(results.map(r => r.promise)), [ 'result', 'result', 'result' ]); + + // A call after the function completes gets the same result. + const later = once.run('a', fn); + assert.equal(later.first, false); + assert.equal(await later.promise, 'result'); + assert.equal(calls, 1); +}); + +test('each key runs its own function', async () => { + const once = new OncePerKey(); + const a = once.run('a', async () => 1); + const b = once.run('b', async () => 2); + assert.equal(a.first, true); + assert.equal(b.first, true); + assert.equal(await a.promise, 1); + assert.equal(await b.promise, 2); +}); + +test('a rejection goes to all callers of the key', async () => { + const once = new OncePerKey(); + const fn = async () => { throw new Error('failed'); }; + const first = once.run('a', fn); + const second = once.run('a', fn); + await assert.rejects(first.promise, /failed/); + await assert.rejects(second.promise, /failed/); +}); diff --git a/test/unit/purge.test.js b/test/unit/purge.test.js index 78de99e..1e5a2b9 100644 --- a/test/unit/purge.test.js +++ b/test/unit/purge.test.js @@ -10,7 +10,7 @@ import path from 'node:path'; import { after, before, beforeEach, describe, test } from 'node:test'; import { loadApiToken } from '../../build/cli/cli/util/api-token.js'; -import { loadPurgeTarget, parsePurgeEnvironments } from '../../build/cli/cli/util/purge.js'; +import { loadPurgeTarget, parsePurgeEnvironments, purgeCollection } from '../../build/cli/cli/util/purge.js'; import { loadServiceId } from '../../build/cli/cli/util/service-id.js'; import { mockConsoleLog } from './mock-console.js'; @@ -127,7 +127,7 @@ describe('loadPurgeTarget', () => { writeFastlyToml('from-toml'); assert.equal(loadPurgeTarget(params({ localMode: true, fastlyApiToken: 'token' })), null); assert.deepEqual(messages(), [ - '- Local mode: will skip purge step after publish.', + '- Local mode: will skip the purge.', ]); }); @@ -136,7 +136,7 @@ describe('loadPurgeTarget', () => { writeFastlyToml(null); assert.equal(loadPurgeTarget(params({ fastlyApiToken: 'token' })), null); assert.deepEqual(messages(), [ - '- Service ID not found (--fastly-service-id, FASTLY_SERVICE_ID, or fastly.toml). Will skip purge step after publish.', + '- Service ID not found (--fastly-service-id, FASTLY_SERVICE_ID, or fastly.toml). Will skip the purge.', ]); }); @@ -171,3 +171,44 @@ describe('loadPurgeTarget', () => { assert.throws(() => loadPurgeTarget(params({ localMode: true, purgeEnvironments: [ 'prod' ] })), /Unknown --purge-environment/); }); }); + +describe('purgeCollection', () => { + const purgeTarget = { + serviceId: 'service', + fastlyApiContext: { apiToken: 'token' }, + environments: [ 'production', 'staging' ], + }; + + test('soft purges the surrogate key of the collection in each environment', async (t) => { + const messages = mockConsoleLog(t); + const fetchMock = t.mock.method(globalThis, 'fetch', async () => new Response('{"status":"ok"}')); + await purgeCollection(purgeTarget, 'site', 'preview'); + const requests = fetchMock.mock.calls.map(call => call.arguments[0]); + assert.deepEqual(requests.map(r => [ + r.method, + r.url, + r.headers.get('surrogate-key'), + r.headers.get('fastly-soft-purge'), + r.headers.get('fastly-purge-environment'), + ]), [ + [ 'POST', 'https://api.fastly.com/service/service/purge', 'site-preview', '1', null ], + [ 'POST', 'https://api.fastly.com/service/service/purge', 'site-preview', '1', 'staging' ], + ]); + assert.deepEqual(messages(), [ + 'Purging surrogate key [site-preview] on service [service] (production)...', + 'Purged', + 'Purging surrogate key [site-preview] on service [service] (staging)...', + 'Purged', + ]); + }); + + test('warns, and does not throw, if a purge fails', async (t) => { + mockConsoleLog(t); + const warn = t.mock.method(console, 'warn', () => {}); + t.mock.method(globalThis, 'fetch', async () => new Response('{}', { status: 403 })); + await purgeCollection({ ...purgeTarget, environments: [ 'production' ] }, 'site', 'preview'); + assert.deepEqual(warn.mock.calls.map(call => call.arguments.join(' ')), [ + '⚠️ Warning: Failed purging (production). Cached copies may be served until they expire.', + ]); + }); +});