feat(dash): the KV-cache TTL analysis page and its strategy simulator

[OsherElhadad]

The dashboard half of the KV-cache work. Stacked on #99, which carries the calculation half — this branch adds no policy and no arithmetic of its own, and it targets feat/kv-cache-ttl-cost-model rather than main because kvcache/ does not exist on main yet, so a PR against main would not compile.

What it is

A tab answering three questions over real history: how long a conversation actually stays idle, what the prompt cache is costing at that idle profile, and what a different TTL policy would have cost on the same requests.

Section	Answers
Coverage statement	what this analysis could not answer — rows with no recorded TTL, rows with no cost, single-request conversations, whether the read was capped
Summary tiles	requests, conversations, median/mean idle, 5-minute and 1-hour reuse probability, cache-hit rate as the provider reported it, cost as billed, requests with no successor, median cached prompt
Distribution	the idle histogram (everything past five minutes in the de-emphasis gray) and the survival curve beside it
Grouped views	the same measurement by observed TTL, user, model and time-of-day band (UTC)
Prices	every rate the simulation uses, editable, with what each comes to on the window's own median prefix
Strategies	the arm picker built from the server's registry, and the comparison table
Detail table	the derived dataset, sortable on 13 columns, paged on the server, each row linking back to the request drawer and the session diff
Formulas	printed from the payload, not retyped in the page

Four GET routes

/api/kvcache, /api/kvcache/rows, /api/kvcache/simulate, /api/kvcache/pricing — all scopeTenant, all returning numbers, enum labels and ids only. No prompt text, no transcript.

GET rather than POST for two reasons worth more than tidiness: both scoping tests probe the mounted route table with a GET, so a POST route would be one neither could check; and a simulation is a view, so its whole input belongs in a URL that can be bookmarked and pasted into an issue.

Not manager-only. The tab carries no data-manager, so every signed-in account sees it scoped to its own traffic. A manager additionally gets the service-wide view — the User filter, the drill-down on By user, and the User column. TestAPIScopesEveryRouteToTheCaller, TestAPIIgnoresCraftedTenantParam and TestAPIFailsClosedWithoutAPrincipal all walk these four routes.

The derivation, and the one thing it would be easy to get wrong

A filter is applied in two places, deliberately. The predicates selecting which conversations are in scope — tenant, time window, session, the exclusion of ping rows — run inside the window function. Everything selecting which requests to show runs outside it. Running model = X inside the partition would make a request's successor the next request on that model, which is not the next request in the conversation, and on this corpus that is not a corner case: it would distort the 12,035 requests sitting in a session that uses more than one model.

The window partitions by (tenant_id, session_id, model), which is exactly kvcache.Conversation. The model is in the key because a cache entry does not transfer between models — an opus request cannot read a sonnet request's entry. TestTheSQLPartitionIsExactlyTheConversationKey asserts the SQL grouping and the Go type agree, structurally and behaviourally, so they cannot drift again.

Three things the page refuses to do

Each is enforced by a test rather than by review, because all three are invisible on screen when wrong — the page looks completely fine.

1. An absence is never rendered as a zero. A request with no successor has idle_ms: null, not 0, and is excluded from every average — 620 of 3,391 on the demo window, and 1,551 of 1,772 conversations on the production corpus hold a single request. A tier that was never recorded reads not recorded in both the grouped table and the row pill, never the tier it is replayed as. A row with incomplete accounting has an unknown cost.
2. Hit rate is not presented as the objective, with a banner saying so. fixed-1h has a better hit rate than the baseline and costs more, so a reader scanning for the best-looking column would pick the worst arm. Nothing sorts or colours by it.
3. A comparison is never clamped, and says which way it points in words — $X cheaper / $X MORE. A column of signed dollars headed "saving" was read as a saving when it was the opposite.

Evidence

Rendered in Chromium against a seeded 3,391-request window shaped like the live corpus (94.9% of gaps inside five minutes against the measured 95.3%; median cached prompt 133k tokens against 124.8k) and inspected panel by panel. That found four defects no substring test could see:

• the exact ceiling reporting the same total as the no-cache arm, because the price list was dropped when threading the config into NewStrategy — every action then cost zero and the DP chose "expire" for everything, presented as the cheapest plan that exists;
• the by-TTL table folding 295 not-recorded rows into the five-minute group and labelling it "configured", while the coverage banner directly above reported them as not recorded;
• the row pill printing 5m for those same rows, with the truth only in a tooltip;
• the formulas panel rendering eleven headings above eleven empty boxes, because the payload emitted {name, expression, prose} while the page read {name, formula, note} — name matched by luck, so it read as a stylesheet problem.

Each is now guarded. The wire contract is checked in both directions from the struct tags, and its comment states what that check does not catch — a field moving between shapes — with the reproduction, rather than letting a green line imply coverage it does not have. Cost identities and the percentage-versus-fraction case are asserted as invariants, because every nominal check is blind to a key that keeps its name and changes its meaning.

Also

Filter gains a TTL dimension with "none" as a sentinel, for the same reason Reason has "compacted": an uncached request's stored value is the empty string, which is also what "no filter" looks like, so an empty field cannot mean it.

Docs: docs/dashboard-kvcache-page.md, in the mkdocs nav, with the four routes and the ttl filter added to docs/reference/routes.md.

Verification

gofmt -l, go vet ./... and go test ./... clean across every package; production binary builds with CGO_ENABLED=1. 51 KV-cache tests in package dash.

Reviewing

Open at /dashboard/#kvcache. Worth checking by hand: that the not recorded TTL group and the coverage banner agree, that optimal reads as a ceiling rather than an option, and that the vs baseline column is unambiguous about direction.