Skip to content

Fix Bedrock Astra caching surface guidance #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
sernote merged 1 commit into from
Oct 22, 2026
Merged

Fix Bedrock Astra caching surface guidance #46

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions audit-prompt-caching/references/bedrock.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ The native Converse response uses lower-camel usage fields: `inputTokens`, `cach
Model contracts differ:
- Anthropic Claude: `cachePoint: {"type": "default", "ttl": "5m" | "1h"}` in `tools`, `system`, and `messages`; up to 4 checkpoints; longer-TTL points must precede shorter ones. The minimum is evaluated on cumulative tokens across `tools` -> `system` -> `messages`, and changing `tools` invalidates the later sections. Minimums are model-tiered (512 for the Claude 5 Opus/Fable/Mythos tier, 1,024 or 4,096 for others at the last review); Bedrock also looks back about 20 content blocks from a breakpoint for a hit. 1h TTL is unavailable on the oldest supported Claude models.
- Amazon Nova: implicit caching for all text prompts plus explicit `cachePoint` in `system` and `messages` only (no `tools`), 5m TTL, about 1K minimum, 4 checkpoints, and a documented 20K-token cap on cacheable content.
- Listed OpenAI GPT-5.6 models and GPT-6 Astra: caching only via the Responses API, on both `bedrock-runtime` (`/openai/v1`, cross-region `us.`/`global.` profiles only) and `bedrock-mantle`. `prompt_cache_breakpoint`/`prompt_cache_options` follow the OpenAI contract with a 30m TTL, 1,024 minimum, 4 writes max, and paid writes (about 1.25x input) with reads at about 0.1x; the usage object is OpenAI-shaped (`usage.input_tokens_details.cached_tokens` and `cache_write_tokens`) and **inclusive**, so do not apply the Converse additive rule there. GPT-5.5 and older are implicit-only with free writes; GPT-OSS models list no caching. As of 2026-09-22, AWS's model cards list GPT-6 Astra but not GPT-6 Sol; verify the exact model ID before applying this contract to a new GPT-6 route.
- OpenAI GPT-6.1 Sol ([model card](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-1-sol.html)) has its own exact-model surface and placement contract; do not copy the GPT-5.6/Astra lists above:
- Listed OpenAI GPT-5.6 models: caching only via the Responses API, on both `bedrock-runtime` (`/openai/v1`, cross-region `us.`/`global.` profiles only) and `bedrock-mantle`. `prompt_cache_breakpoint`/`prompt_cache_options` follow the OpenAI contract with a 30m TTL, 1,024 minimum, 4 writes max, and paid writes (about 1.25x input) with reads at about 0.1x; the usage object is OpenAI-shaped (`usage.input_tokens_details.cached_tokens` and `cache_write_tokens`) and **inclusive**, so do not apply the Converse additive rule there. GPT-5.5 and older are implicit-only with free writes; GPT-OSS models list no caching.
- OpenAI GPT-6.1 Sol ([model card](https://docs.aws.amazon.com/bedrock/latest/userguide/model-card-openai-gpt-6-1-sol.html)) has its own exact-model surface and placement contract; do not copy the GPT-5.6 list above. GPT-6 Astra, per its own card, shares the bullets below except Routes (its routes and tiers differ):
- `bedrock-runtime`: implicit caching on Responses, Chat Completions, InvokeModel, and Converse; explicit caching on Responses, Chat Completions, and InvokeModel. Converse is implicit-only for this model: flag a native `cachePoint` as unsupported and audit prefix stability instead. `bedrock-mantle` (`/openai/v1`): implicit and explicit on Responses and Chat Completions.
- Explicit: set `prompt_cache_options.mode` to `explicit` (the default `implicit` mode also adds an automatic latest-message breakpoint; `explicit` with no breakpoint does not cache) and `prompt_cache_options.ttl` to `30m`, the only supported TTL and the default, so flag any other value. AWS separately says cached prefixes stay reusable for at least 30 minutes, so `30m` is minimum retention, not an exact eviction deadline: a longer gap is a possible-miss risk that proves neither zero reads nor per-request billable writes; check raw reads/writes and actual retention/cadence before a cold-start or cost claim. Add `"prompt_cache_breakpoint": {"mode": "explicit"}` to an `input_text` block in Responses or a `text` content part in Chat Completions and InvokeModel; GPT-5.6's `input_image`/`input_file` placement is not listed for Sol.
- The cumulative prefix before a breakpoint must reach 1,024 tokens. Multiple explicit breakpoints are allowed, and AWS states each request can create up to four cache writes; keep that wording rather than inventing a breakpoint-count rule.
Expand Down
Loading