Skip to content

fix: preserve provider prompt cache (date-only timestamps + append at end) - #25

Open
caos30 wants to merge 3 commits into
joshuadavidthomas:mainfrom
caos30:main
Open

fix: preserve provider prompt cache (date-only timestamps + append at end)#25
caos30 wants to merge 3 commits into
joshuadavidthomas:mainfrom
caos30:main

Conversation

@caos30

@caos30 caos30 commented Aug 19, 2026

Copy link
Copy Markdown

fix: preserve provider prompt cache (date-only timestamps + append at end)

Problem

The system-prompt injection busts the provider KV/prompt cache prefix on every request, even when no memory block changes. Two independent causes:

  1. ms-precision timestampsrenderMemoryMetadata() embeds now.toISOString() and per-block lastModified at millisecond granularity, so the injected XML differs on every single request. (<memory_metadata> breaks KV-cache / prompt-cache by injecting ms-precision timestamp every turn #24)
  2. Volatile content spliced at position 1output.system.splice(1, 0, xml) places the memory XML before nearly the entire cacheable prefix (rest of system prompt, tools, conversation history). Any memory edit invalidates the whole cache, not just the memory segment. (System prompt injection busts Anthropic prompt cache on every turn #8)

Users on Anthropic/OpenAI/DeepSeek-style billing see cache hit ratios near zero with this plugin active (measured ~10% where ~90% is expected on long sessions).

Fix

  • Truncate both timestamps to YYYY-MM-DD — the XML is now byte-stable across requests within a day, and across the common case of memory_set/memory_replace edits that don't change the newest-block date.
  • Append the memory XML at the end of the system prompt instead of splicing at index 1. A memory edit now only invalidates the small trailing segment; the large stable prefix (provider header, tools, instructions, history) stays cacheable.

Both changes preserve existing behaviour: same XML content, same blocks, same journal handling (journal note still appended last).

Note: PR #20 addresses cause 1 only; this PR covers both. Happy to rebase/split if preferred.

Testing

  • bun test: 27 pass / 0 fail
  • Smoke-tested the rendered <memory_metadata>: no sub-day timestamp present.

Fixes #24. Refs #8.

caos30 added 2 commits August 19, 2026 19:21
The ms-precision ISO timestamps (`now.toISOString()`, per-block
`lastModified`) made the injected XML differ on every request, so the
provider KV/prompt cache prefix was invalidated each turn even when no
memory block changed. Truncate both to YYYY-MM-DD granularity.

Fixes joshuadavidthomas#24, refs joshuadavidthomas#8
Splicing the memory XML at position 1 placed volatile content before
nearly the entire cacheable prefix (system prompt, tools, history), so
any memory edit busted the whole provider prompt cache. Append at the
end instead: edits now only invalidate the small trailing segment.

Refs joshuadavidthomas#8
@assert-app

assert-app Bot commented Aug 19, 2026

Copy link
Copy Markdown

Review on Assert →

2 clusters identified

@caos30

caos30 commented Aug 19, 2026

Copy link
Copy Markdown
Author

I’ve been using this OpenCode plugin for almost three months, and I’m in love with it and with how it helps my agent truly learn about me, my environment, and my projects in every session. Thanks for such great work!

I submitted this PR because my GLM-5.3 agent and I discovered that my sessions had almost zero cache hits. It was able to diagnose and fix the issue, so I think this is important for other users as well.

I usually use “token plans” with hourly or weekly quotas, but when I use PAYG providers such as DeepSeek, Mistral, or NeuralWatt, my purchased credits disappear incredibly quickly! 😔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

<memory_metadata> breaks KV-cache / prompt-cache by injecting ms-precision timestamp every turn

1 participant