feat(dash): the KV-cache TTL analysis page and its strategy simulator

[OsherElhadad]

Re-targets #104 at main. #104 was merged into the wrong place and its content never reached main — this PR is the same commit, rebased onto main, and it is the one to merge.

What went wrong with #104

#99 (the calculation half) was squash-merged into main at 19:21. #104 was based on feat/kv-cache-ttl-cost-model and merged into that branch at 19:34 — thirteen minutes after the branch had stopped being on any path to main. So the dashboard commit sits on the feature branch and main never got it, which is why the tab does not appear.

I based #104 on the feature branch because kvcache/ did not exist on main at the time, so a PR against main would not have compiled. That reasoning was right when I opened it and stale by the time it merged. The correct move would have been to wait for #99 and target main, or to retarget #104 the moment #99 landed.

Now that #99's content is on main, kvcache/ is there and this compiles against it. The cherry-pick applied with no conflicts, and everything below was verified on top of main (7b20027), not on the old base.

What it is

The dashboard half of the KV-cache work: a tab answering how long a conversation actually stays idle, what the prompt cache costs at that idle profile, and what a different TTL policy would have cost on the same requests. The policies, the cost model and the replay are package kvcache — nothing here decides or prices anything.

Section	Answers
Coverage statement	what the analysis could not answer — no recorded TTL, no cost, single-request conversations, whether the read was capped
Summary tiles	requests, conversations, median/mean idle, 5-minute and 1-hour reuse probability, cache-hit rate as the provider reported it, cost as billed, requests with no successor, median cached prompt
Distribution	the idle histogram (past five minutes in the de-emphasis gray) and the survival curve
Grouped views	the same measurement by observed TTL, user, model, time-of-day band (UTC)
Prices	every rate the simulation uses, editable, with what each comes to on the window's own median prefix
Strategies	the arm picker built from the server's registry, and the comparison table
Detail table	the derived dataset, sortable on 13 columns, paged on the server, each row linking back to the request drawer and the session diff
Formulas	printed from the payload, not retyped in the page

Four GET routes

/api/kvcache, /api/kvcache/rows, /api/kvcache/simulate, /api/kvcache/pricing — all scopeTenant, returning numbers, enum labels and ids only. No prompt text, no transcript.

GET rather than POST for two reasons worth more than tidiness: both scoping tests probe the mounted route table with a GET, so a POST route would be one neither could check; and a simulation is a view, so its whole input belongs in a URL that can be bookmarked and pasted into an issue.

Not manager-only. The tab carries no data-manager, so every signed-in account sees it scoped to its own traffic. A manager additionally gets the service-wide view — the User filter, the drill-down on By user, and the User column. TestAPIScopesEveryRouteToTheCaller, TestAPIIgnoresCraftedTenantParam and TestAPIFailsClosedWithoutAPrincipal all walk these four routes.

The derivation, and the thing it would be easy to get wrong

A filter is applied in two places, deliberately. The predicates selecting which conversations are in scope — tenant, time window, session, the exclusion of ping rows — run inside the window function. Everything selecting which requests to show runs outside it. Running model = X inside the partition would make a request's successor the next request on that model, which is not the next request in the conversation, and that would distort the 12,035 requests sitting in a session that uses more than one model.

The window partitions by (tenant_id, session_id, model), which is exactly kvcache.Conversation. The model is in the key because a cache entry does not transfer between models — an opus request cannot read a sonnet request's entry, and linking them grants the second a hit at 0.1× on an entry it could never have matched, one-directionally making every arm look cheaper and hit more often than it can. TestTheSQLPartitionIsExactlyTheConversationKey asserts the SQL grouping and the Go type agree structurally and behaviourally, so they cannot drift again — this bug was live until the pre-commit test run caught it.

Three things the page refuses to do

Each enforced by a test rather than by review, because all three are invisible on screen when wrong.

1. An absence is never rendered as a zero. A request with no successor has idle_ms: null, not 0, and is excluded from every average. A tier never recorded reads not recorded in both the grouped table and the row pill, never the tier it is replayed as. A row with incomplete accounting has an unknown cost.
2. Hit rate is not presented as the objective, with a banner saying so: fixed-1h has a better hit rate than the baseline and costs more, so a reader scanning for the best-looking column would pick the worst arm. Nothing sorts or colours by it.
3. A comparison is never clamped and says which way it points in words — $X cheaper / $X MORE.

Evidence

Rendered in Chromium against a seeded 3,391-request window shaped like the live corpus (94.9% of gaps inside five minutes against the measured 95.3%; median cached prompt 133k tokens against 124.8k) and inspected panel by panel. That found four defects no substring test could see: the exact ceiling reporting the same total as the no-cache arm because the price list was dropped when threading the config; the by-TTL table folding 295 not-recorded rows into the five-minute group while the coverage banner directly above reported them as not recorded; the row pill printing 5m for those same rows; and the formulas panel rendering eleven headings above eleven empty boxes because the payload emitted {name, expression, prose} while the page read {name, formula, note}. Each is now guarded.

The wire contract is checked in both directions from the struct tags, and its comment states what that check does not catch — a field moving between shapes — with the reproduction, rather than letting a green line imply coverage it lacks. Cost identities and the percentage-versus-fraction case are asserted as invariants, because every nominal check is blind to a key that keeps its name and changes its meaning.

Also

Filter gains a TTL dimension with "none" as a sentinel, for the same reason Reason has "compacted": an uncached request's stored value is the empty string, which is also what "no filter" looks like.

Docs: docs/dashboard-kvcache-page.md, in the mkdocs nav, with the four routes and the ttl filter added to docs/reference/routes.md.

Verification, on top of main

gofmt -l, go vet ./... and go test ./... clean across every package; CGO_ENABLED=1 production binary builds. 51 KV-cache tests in package dash.

After merging

The UI is go:embed-ed into the binary, so a merge alone changes nothing on a running service: it needs a rebuild and a restart, and the dashboard needs -dashboard / DASHBOARD=1. Open at /dashboard/#kvcache.