You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Suggested priority: P2 — reduce repeat-turn time to first token.
The server resets model state and prefills the whole conversation for each request. Add an explicit session API that retains compatible KV state and processes only newly appended tokens, with clear invalidation and resource limits.
Scope and acceptance criteria
Define session lifecycle, reset/dispose behavior, and maximum retained-session memory.
Match the actual tokenized prefix, including BOS and chat-template tokens; do not infer reuse from raw text alone.
Reuse valid KV positions for append-only conversation turns and prefill only the suffix.
Invalidate or recompute state on model/adapter/template changes, edited history, context eviction, or incompatible settings.
Keep session histories isolated; require explicit opt-in for server session retention and avoid cross-client cache reuse by default.
Support browser conversations and a documented server integration without silently changing stateless request semantics.
Persist conversation records if requested; clearly distinguish that from persisting GPU/KV state, which can be rebuilt after reload.
Compare cached versus full-prefill logits/output for greedy generation and cover edits, cancellation, eviction, and reset.
Benchmark multi-turn TTFT and memory at several context lengths, reporting how many tokens were reused.
Related: closed #99 added conversation history to the demo; this issue adds compute-state reuse rather than just retaining messages. Depends on a correct prefill-to-decode transition.
Suggested priority: P2 — reduce repeat-turn time to first token.
The server resets model state and prefills the whole conversation for each request. Add an explicit session API that retains compatible KV state and processes only newly appended tokens, with clear invalidation and resource limits.
Scope and acceptance criteria
Evidence: server reset and generation, browser reset, KV cache.
Related: closed #99 added conversation history to the demo; this issue adds compute-state reuse rather than just retaining messages. Depends on a correct prefill-to-decode transition.