Skip to content

feat(ingress): map enable_thinking/reasoning_effort onto the thinking object - #437

Merged
jsboige merged 2 commits into
mainfrom
fix/435-ingress-thinking-controls
Oct 10, 2026
Merged

jsboige merged 2 commits into
mainfrom
fix/435-ingress-thinking-controls

Conversation

@jsboige

@jsboige jsboige commented Oct 10, 2026

Copy link
Copy Markdown
Owner

What

The OpenAI ingress (/v1/chat/completions) translator silently dropped enable_thinking, chat_template_kwargs.enable_thinking and reasoning_effort — the reasoning decision fell to the served dialect's policy (CLAUDISH_GLM_THINKING passthrough default = GLM thinks; CLAUDISH_QWEN_THINKING disabled) and the client's ask never reached prepareRequest. They now map onto the house thinking object, the seam every dialect already consults (originalRequest.thinking).

Client field Mapped shape
enable_thinking: false {type: "disabled"}
enable_thinking: true {type: "enabled", budget_tokens: 2048}
reasoning_effort: minimal/low/medium/high {type: "enabled", budget_tokens: 2000/8000/24000/32000}
  • Top-level enable_thinking wins over the chat_template_kwargs spelling; unknown effort value → ignored (no partial mapping); no controls → unchanged behavior.
  • The effort ladder is the exact inverse of openai-api-format.ts's budget→effort map, so the o1/o3 round trip is stable.
  • Budget legality: enabled blocks always carry budget_tokens ≥ 1024 (native Anthropic floor) and < max_tokens (Qwen requirement). Against a client-sent max_tokens the budget clamps into it; against our own 4096 placeholder it raises the placeholder to budget + 1 — clamping "high" (32000) to 4095 would invert the ask to "minimal" downstream. No room → mapping skipped, dialect policy decides.

Second defect found and fixed in passing: the Qwen openai-wire passthrough mapped ANY originalRequest.thinking to enable_thinking: true — the exact inversion of an explicit disabled ask arriving from this new mapping.

Acceptance criteria

AC1 — per-field pin at the translated thinking: 11 pins in openai-request-to-anthropic.test.ts (disabled ×2 spellings + precedence, enabled default, effort ladder, placeholder-raise, client-ceiling clamp, no-room skip, unknown-effort, no-controls, non-mutation) + 4 Qwen dialect interaction pins + new glm-model-dialect.test.ts suite (4 pins, with the 2026-08-20 probe of record in its header).

AC2 — real-lane control, no budget spent beyond the GLM Coding plan. Probe chain: branch-build proxy (sandboxed config, port 3980) → customEndpoint hubprobe (anthropic format) → hub /v1/messages → sonnet nominal gc@glm-5.3 — so the hub's GLM dialect emits the field on the openai wire. Prompt "Reponds exactement: ok", max_tokens: 200, 2026-10-11:

Run completion reasoning_content content
A — no thinking controls 36 131 chars "ok"
B — enable_thinking: false 3 absent "ok"

The 37→3 shape of the 2026-08-20 probe of record, end to end through the ingress.

AC3 — doc lists the mapped request fields: docs/reference/openai-ingress.md gains a "#435 request-side thinking controls" section (table + budget rules + downstream fates per dialect + the measurement), and the request-converter table now lists the three fields. CLAUDE.md ingress section carries the decision-level line.

Verification

  • bun test on the four suites (translator, qwen dialect, glm dialect, route-level proxy-server-openai-notice.test.ts): 51/51 pass.
  • Scoped tsc (-p packages/cli/tsconfig.json): 226 pre-change = 226 post-change (verified by stashing the working tree and re-running) — zero new errors; the two TS6133 in dialect files predate this change.

Closes #435

🤖 Generated with Claude Code

jsboige and others added 2 commits October 11, 2026 01:03
… object (#435)

The OpenAI ingress translator silently dropped `enable_thinking`,
`chat_template_kwargs.enable_thinking` and `reasoning_effort`, so the
reasoning decision fell to the served dialect's policy and the client's
ask never reached prepareRequest. Map them onto the house `thinking`
object — the seam every dialect already consults:

- enable_thinking:false -> {type:"disabled"} (the GLM 37->3 probe shape)
- enable_thinking:true  -> enabled, 2048 default budget
- reasoning_effort      -> inverse of openai-api-format's budget ladder
  (2000/8000/24000/32000), so the o1/o3 round trip is stable

An enabled block is always legal (>=1024, < max_tokens): clamps against
a client-sent max_tokens; against our own 4096 placeholder it RAISES the
placeholder instead — clamping "high" to 4095 would invert the ask to
"minimal" downstream. No room -> skip, dialect policy decides.

Same fix closes the Qwen openai-wire passthrough, which mapped ANY
originalRequest.thinking to enable_thinking:true — the exact inversion
of an explicit disabled ask.

Measured on the real lane (probe proxy -> hub /v1/messages -> sonnet
nominal gc@glm-5.3, "Reponds exactement: ok"): no controls -> 36
completion tokens / 131 chars reasoning; enable_thinking:false -> 3
tokens, reasoning absent — the 2026-08-20 probe shape, end to end.

Pins: translator mapping matrix (11), Qwen dialect interaction (4),
new GLM dialect suite (4). 51/51 green across the four suites; tsc
delta 0 (verified against the pre-change tree).

Refs #435

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…elsewhere in the file)

Refs #435

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@jsboige
jsboige merged commit 08f6f32 into main Oct 10, 2026
1 check passed
jsboige added a commit that referenced this pull request Oct 11, 2026
…onerated (#438)

#399 measured (07/10, hub + vLLM frognano-4b :5003) reasoning on every
hub-served request while direct :5003 never thinks. Investigation
(2026-10-11), all measured:

- The paired inbound capture req-1-11725 (19:39:33Z) is MINIMAL — no
  thinking, no tools, no system. The outbound body built for it was
  captured through the real pipeline via an echo endpoint: byte-identical
  198 bytes at 6f732eb (main on 07/10) and af52be5 (current) — model,
  messages, temperature, stream, stream_options, max_tokens. NO thinking
  field of any spelling.
- frognano-4b matches neither qwen nor alibaba (matchesModelFamily =
  startsWith | "/fam"), so QwenModelDialect — whose passthrough could
  emit enable_thinking — never ran for it. The #437-passthrough
  hypothesis is refuted for this issue.
- The flip no longer reproduces via the hub (11/10): the exact r11725
  body AND a 26-tools consumer body (req-1-6396) both answer immediately
  with no reasoning. Same body both eras => the change was server-side
  (:5003 deployment, refs vllm#70/#71).

Pins: the six neutral fields exactly; no thinking field even with an
inbound enabled thinking block off the o1/o3 lane; o1 keeps the
budget->reasoning_effort mapping (the only emission site).

Refs #399

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
@jsboige

jsboige commented Oct 11, 2026

Copy link
Copy Markdown
Owner Author

Suivi de la note non bloquante du merge — pourquoi le placeholder 32001 ne reçoit PAS de clamp supplémentaire (argument écrit, grain 3 deepqueue 11/10).

La note : sans max_tokens client, reasoning_effort: "high" porte le placeholder à 32001, et rien ne plafonne max_tokens au plafond de sortie du provider. Trois points, lus dans le code :

  1. Quand un plafond EXISTE, il est déjà appliqué — avec le budget. Le clamp maxOutputTokens de l'endpoint (composed-handler.ts:543, feat(failover): a cascade step that cannot take a request is skipped, not overflowed or queued (Swift as a Haiku step) #431) borne max_tokens — 32001 inclus — et tire le budget_tokens à cap-1 pour que la paire reste légale sur le fil Anthropic. C'est pinné : K7 (proxy-server-step-skip-route.test.ts :500, « maxOutputTokens clamps max_tokens on the wire — openai and anthropic formats ») avec sa mutation nommée m3 (« remove the clamp ⇒ K7 red »). Un endpoint qui déclare son plafond est couvert bout en bout.

  2. Quand AUCUN plafond n'est déclaré, il n'y a rien à respecter. L'ingress ne connaît pas le provider au moment de la traduction (le routage — et donc l'endpoint — résout UNE couche plus tard). Caper à une constante arbitraire (4096 ?) ré-inverserait l'ask (« high » → « minimal » au remap o1/o3) — exactement le défaut que la règle placeholder-raise existe pour prévenir. « Pas de plafond déclaré » est la déclaration de la deployment ; la traductrice n'a pas à l'outrepasser.

  3. Résidu assumé, bruyant plutôt que silencieux. Sur une lane native à petit plafond de sortie, 32001 peut 400 (invalid_request_error) — mais l'échec est retentissant et retryable (le client renvoie avec un max_tokens explicite), pas une inversion silencieuse de son ask. Donner au translateur une dépendance au catalogue de modèles déplacerait la connaissance modèle dans la mauvaise couche ; c'est le refus conscient du point 2.

Conclusion : aucun changement de code — le clamp correct vit déjà à la couche qui possède la connaissance du plafond, le cas sans plafond est indécidable à la couche ingress, et le résidu échoue haut et clair. Si un endpoint réel 400 sur cette forme, le geste est de déclarer maxOutputTokens sur CET endpoint (le levier existe, il est testé).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

openai-ingress: request-side thinking controls (enable_thinking, chat_template_kwargs, reasoning_effort) are silently dropped at translation

1 participant