Repository navigation
fix: keep finish_reason in create_stream when the finish chunk also carries usage - #8278
Open
Lucas Kim (kimnamu) wants to merge 1 commit into
Open
Lucas Kim (kimnamu) wants to merge 1 commit into
Lucas Kim (kimnamu) wants to merge 1 commit into
Conversation
…arries usage OpenAIChatCompletionClient.create_stream only read choice.finish_reason from chunks without usage. Servers that attach usage to the finish chunk (e.g. the Amazon Bedrock runtime /openai/v1 endpoint with GPT-6 models) always ended up with finish_reason "unknown", including length truncation and tool calls. The existing `stop_reason is None` guard already keeps an earlier finish_reason from being overwritten by a trailing usage chunk. Co-Authored-By: Claude <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Thanks for keeping
create_streamworking across so many OpenAI-compatible servers; the usage-chunk handling from #3972 is what this builds on.create_streamreadsfinish_reasononly from chunks withusage is None(_openai_client.pyL948). The Amazon Bedrock runtime endpoint/openai/v1/chat/completionsattaches usage to the GPT-6 finish chunk, so the streamedCreateResult.finish_reasonis always"unknown", even forlengthand tool calls, whilecreate()on the same request returns the real value.This drops the
chunk.usage is Nonecondition. Thestop_reason is Nonecheck that stays already keeps an earlier finish reason when a LiteLLM-style usage chunk follows. The Azure AI client already readsfinish_reasonwhenever it is set.us.openai.gpt-6-sol/us.openai.gpt-6-luna(andglobal.) stream: text, truncated, tool call¹unknown×3 eachstop/length/function_calls(same ascreate())us.openai.gpt-6-astra/global.openai.gpt-6-astrastream: text, truncatedunknown×2 eachstop/length(same ascreate())openai.gpt-oss-120b-1:0, Mantleopenai.gpt-6-sol/openai.gpt-6-astra(usage not on finish chunk)stop(existing tests)¹ with
reasoning_effort="none", which Bedrock chat completions needs for Sol/Luna function tools.Test RED/GREEN, checks, live runs
New test
test_openai_chat_completion_client_create_stream_finish_reason_with_usage(mocked: content chunk → finish chunk with usage →choices=[]usage chunk,stopandlength):tests/models/test_openai_model_client.py: main 2 failed / 53 passed (the new test), fix 55 passed, 33 skipped. The agentchat tests that use this client (test_assistant_agent.py,test_group_chat.py,test_declarative_components.py): 177 passed, 3 skipped.ruff format --checkandruff checkclean,pyright0 errors,mypyno issues on both files.Live,
base_url=https://bedrock-runtime.us-east-1.amazonaws.com/openai/v1,max_completion_tokens=5:Astra has no tool-call row: Bedrock's chat completions endpoint returns 400 for function tools with Astra ("To use function tools, use /v1/responses"), before and after this change.
Related issue number
Fixes #8277
Checks