Skip to content

Add persistent chat sessions with safe KV prefix reuse #525

Description

@sauravpanda

Suggested priority: P2 — reduce repeat-turn time to first token.

The server resets model state and prefills the whole conversation for each request. Add an explicit session API that retains compatible KV state and processes only newly appended tokens, with clear invalidation and resource limits.

Scope and acceptance criteria

  • Define session lifecycle, reset/dispose behavior, and maximum retained-session memory.
  • Match the actual tokenized prefix, including BOS and chat-template tokens; do not infer reuse from raw text alone.
  • Reuse valid KV positions for append-only conversation turns and prefill only the suffix.
  • Invalidate or recompute state on model/adapter/template changes, edited history, context eviction, or incompatible settings.
  • Keep session histories isolated; require explicit opt-in for server session retention and avoid cross-client cache reuse by default.
  • Support browser conversations and a documented server integration without silently changing stateless request semantics.
  • Persist conversation records if requested; clearly distinguish that from persisting GPU/KV state, which can be rebuilt after reload.
  • Compare cached versus full-prefill logits/output for greedy generation and cover edits, cancellation, eviction, and reset.
  • Benchmark multi-turn TTFT and memory at several context lengths, reporting how many tokens were reused.

Evidence: server reset and generation, browser reset, KV cache.

Related: closed #99 added conversation history to the demo; this issue adds compute-state reuse rather than just retaining messages. Depends on a correct prefill-to-decode transition.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or improvementperformancePerformance optimization

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions