feat(gateway): llm.disableThinking — inject thinking={type:disabled} into chat-completions bodies - #899
Open
francialisomlimoeiro wants to merge 1 commit into
Conversation
…into chat-completions bodies
Some reasoning models served via OpenAI-compatible endpoints spend the
entire max_tokens budget on reasoning tokens unless thinking is
explicitly disabled; the gateway's structured (JSON) extraction then
gets truncated to nothing.
Add an opt-in disableThinking flag:
- standalone adapter (llm-runner): when set, chat-completions requests
go through a fetch wrapper that injects thinking={type:disabled} into
every JSON body carrying a messages array. This is needed because
@ai-sdk/openai does not forward providerOptions into the
OpenAI-compatible request body.
- gateway config: yaml llm.disableThinking / env TDAI_LLM_DISABLE_THINKING.
- gateway offload LLM path: same injection for consistency.
Config:
llm:
disableThinking: true
Adds src/adapters/standalone/thinking-disabled-fetch.test.ts (4 tests:
chat-body injection, non-chat and non-JSON bodies pass through, bodyless
requests untouched).
Collaborator
|
Thanks for your contribution and attention! We will review this PR and get back to you as soon as possible. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What it does
Some reasoning models served via OpenAI-compatible endpoints spend the entire
max_tokensbudget on reasoning tokens unless thinking is explicitly disabled; the gateway's structured (JSON) extraction then gets truncated to nothing.@ai-sdk/openaidoes not forwardproviderOptionsinto the OpenAI-compatible request body (AI SDK v6), so injection happens at fetch level.src/adapters/standalone/llm-runner.ts: opt-indisableThinking→ fetch wrapper injectsthinking: {type:"disabled"}into every JSON body carrying amessagesarray; non-chat/non-JSON bodies pass through.src/gateway/config.ts: yamlllm.disableThinking/ envTDAI_LLM_DISABLE_THINKING.src/gateway/server.ts: same injection on the gateway offload LLM path.Tests
New
src/adapters/standalone/thinking-disabled-fetch.test.ts(4 tests).vitest run→ 3 files, 19 passed (incl. repo's existing suites). esbuild transpile check clean.