fix(server): send spec-compliant finish_reason on stream chunks - #1246
Open
MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
Open
fix(server): send spec-compliant finish_reason on stream chunks#1246MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
MAN$I VERMA (mansiverma897993) wants to merge 1 commit into
Conversation
Streaming chunks reused openai-go response structs, whose plain-string FinishReason marshals as "" on every chunk, and no terminal chunk was emitted at all. OpenAI-compatible clients (e.g. AnythingLLM) wait for a final chunk with finish_reason "stop"/"length"/"tool_calls" and never detect completion. Use local chunk types that serialize finish_reason as null on intermediate chunks and close every stream with an empty-delta finishing chunk mapped from the SDK stop_reason; the tool-call fallback path now also terminates with a finishing chunk and [DONE]. Fixes qualcomm#1243 Signed-off-by: mansiverma897993 <vmansi756@gmail.com>
This was referenced Jul 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1243
Root cause
The streaming handlers built SSE chunks from the openai-go response structs (
openai.ChatCompletionChunk/ChatCompletionChunkChoice). TheirFinishReasonfield is a plainstring, so every chunk serialized as"finish_reason": ""— and no terminal chunk was ever emitted beforedata: [DONE]. OpenAI-compatible clients wait forfinish_reason: "stop"on a final empty-delta chunk, so they never detect completion (as reported with AnythingLLM).Fix
Introduced small local chunk types in
cli/server/handler/chat.gowith spec-correct JSON semantics and closed every stream with a proper finishing chunk:"finish_reason": null(pointer field, no omitempty)"delta": {}withfinish_reasonmapped from the SDK stop_reason via the existingmapFinishReason(so max-token truncation correctly yields"length", everything else"stop")finish_reason: null, then a finishing chunk with"tool_calls"; the parse-fallback text path previously ended the stream with no finishing chunk and no[DONE]— it now terminates properly toostream_options.include_usage) keeps its"choices": []shape"object": "chat.completion.chunk"Example final chunk now on the wire:
{"object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}Non-streaming responses are untouched (they already returned
finish_reason: "stop").Testing
cli/server/handler/package_test.goasserting the serialized shapes: nullfinish_reasonon content chunks,"stop"+ empty delta on the finishing chunk, emptychoiceson the usage chunk, and themapFinishReasontable.{"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":"hello"},"finish_reason":null}]}{"object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}geniex serveend-to-end — Novatoris could you verify with AnythingLLM once a build with this patch is available?