[Compatibility] Add an opt-in Responses stream/terminal repair escape hatch for custom providers

[Wibias]

Summary

Custom openai-responses providers can hit compatibility failures where the upstream turn completes but Native Codex does not receive a terminal lifecycle it can commit. PR #1367 reproduced this class of failure and explored a per-model bounded-JSON fallback, but that PR cannot land as written and should be treated as evidence/design exploration rather than an implementation to salvage.

The remaining requirement is still valid: custom Responses providers need a narrow, explicit compatibility escape hatch that uses the current Responses repair architecture instead of adding another broad transport/configuration path.

Why this issue exists

PR #1367 reproduced the following boundary on both macOS arm64 and GNU/Linux x86_64:

• the same Responses upstream completed when Native Codex connected directly;
• through OpenCodex, the upstream/internal path reached response.completed;
• Native Codex did not commit the turn and later entered its stream-idle retry path;
• the reproduction did not require the reporter's separate OCI relay;
• minimal calls could succeed, while richer turns reproduced the delivery failure.

That is useful evidence that there is a real Responses compatibility class after upstream completion and before the client commits the terminal event. It does not prove one specific Bun primitive is the sole cause.

The maintainer triage on #1367 also makes the current direction explicit:

The failure class is real and still uncovered. Current terminal repair is a narrower registry-controlled case, and no custom-provider escape hatch exists on current dev. A redesign around the current registry-only compatibility hint + terminal-repair architecture would be welcome.

PR #1367 itself is conflicting, far behind dev, and has unresolved blockers, so this issue intentionally does not copy its implementation surface.

Current architecture to build on

Current dev already has a registry-only Responses terminal-repair path in:

• src/providers/registry.ts
• src/server/responses-terminal-repair.ts
• src/server/responses/core.ts

That path was designed for the narrow DeepSeek case where output items are structurally complete but a real terminal is missing or indefinitely delayed. It keeps upstream streaming, relays healthy streams unchanged, and can synthesize one terminal only after strict structural checks and a grace period.

This issue should extend that architecture rather than create a second broad Responses validation/reframing subsystem.

Proposed direction

Add a custom-provider, per-model compatibility hint for effective openai-responses routes. The hint should resolve through the same model/wire policy machinery used by registry compatibility policies and should be inactive unless explicitly configured.

The implementation should prefer the smallest repair that matches the observed failure:

1. Terminal repair first. Allow an eligible custom Responses model to opt into the existing bounded terminal-repair state machine, with the same fail-closed structural requirements used by registry-controlled repair.
2. Preserve healthy streams byte-for-byte where possible. A real upstream terminal remains authoritative and must never be duplicated or replaced.
3. Do not silently apply repair to unrelated providers/models. Resolve after the effective per-model wire is known and reject the setting when that wire is not openai-responses.
4. Keep forward-auth/canonical OpenAI routes protected. Do not allow a custom compatibility setting to mutate the canonical forward-auth contract unless a separate design explicitly approves that.
5. If the feat(responses): add an opt-in bounded JSON fallback for custom providers #1367 reproduction remains broken because the upstream terminal is present but is lost after the relay/inspection split, evaluate a bounded non-streaming fallback as a second, explicit compatibility mode. That fallback should live behind the same narrow compatibility-policy boundary. It should not reintroduce feat(responses): add an opt-in bounded JSON fallback for custom providers #1367's broad shared WebSocket validation changes or make non-streaming the default.

A shape such as the following is illustrative, not prescribed:

{
  "providers": {
    "<custom-provider>": {
      "adapter": "openai-responses",
      "modelResponsesCompatibility": {
        "<model>": "terminal-repair"
      }
    }
  }
}

If more than one repair mode is eventually justified, prefer an explicit enum/policy object over multiple overlapping booleans.

Acceptance criteria

• A custom provider model whose effective wire is openai-responses can explicitly opt into the supported repair mode.
• The default behavior of unconfigured providers/models is unchanged.
• Registry-controlled repair continues to work unchanged.
• Healthy SSE with a real response.completed, response.failed, or response.incomplete remains authoritative and is not duplicated.
• Partial, malformed, tainted, cancelled, aborted, unknown-item, or over-budget output never gets fabricated as successful completion.
• Function calls preserve item ids, call ids, names, arguments, ordering, and continuation usability.
• HTTP/SSE and Responses WebSocket behavior have focused regression coverage where the selected repair mode applies.
• Client cancellation aborts/releases the repair state and any upstream read without retained timers or buffers.
• Case-insensitive per-model configuration cannot contain conflicting duplicate keys.
• The policy is validated against the effective per-model wire, including modelAdapters/virtual-model resolution where applicable.
• Diagnostics, if added, remain content-free and do not log prompts, response text, credentials, provider secrets, or raw payloads.
• Live regression coverage includes the failure shape from feat(responses): add an opt-in bounded JSON fallback for custom providers #1367 on macOS arm64 and Linux x86_64 before the issue is considered fixed.

Design guardrails

Do not use this issue to:

• expose the old feat(responses): add an opt-in bounded JSON fallback for custom providers #1367 implementation wholesale;
• globally tighten validation for every successful Responses JSON/WebSocket snapshot;
• force all custom Responses providers to stream:false;
• add provider/model name heuristics;
• change the default streamMode;
• bypass the Bun/runtime safety gate for eager relay;
• replace the broader protocol-safe one-reader work tracked by [architecture][memory] Make 32 concurrent tool-recall sessions protocol-safe and memory-bounded #820;
• claim terminal repair alone fixes the feat(responses): add an opt-in bounded JSON fallback for custom providers #1367 relay-loss reproduction without reproducing it on current dev.

The important distinction is that terminal repair and downstream relay loss are related compatibility symptoms, but they are not automatically the same bug. The implementation should prove which repair mode covers the current reproduction instead of assuming it.

Related

• feat(responses): add an opt-in bounded JSON fallback for custom providers #1367 - original draft and reproduction evidence; do not use as the implementation template
• [architecture][memory] Make 32 concurrent tool-recall sessions protocol-safe and memory-bounded #820 - long-term runtime-qualified, protocol-safe one-reader/stream architecture
• #893 is not fixed on macOS: #1025 repairs snapshots but the transport gate from closed #928 is still missing — Codex hangs on third-party Responses gateways #1127 - prior macOS Responses delivery failure with upstream success and missing client SSE
• src/server/responses-terminal-repair.ts - current bounded terminal-repair state machine
• docs/superpowers/plans/2026-08-06-deepseek-responses-streaming-terminal-repair.md - design rationale for the current registry-only repair path

[coderabbitai]

🔗 Related PRs

#749 - fix(responses): reject expired forward continuations [merged]
#928 - fix(streaming): repair sparse Responses snapshots [open]
#1095 - fix(deepseek): restore progressive Responses streaming [open]
#1361 - fix(responses): restore DeepSeek streaming and routed code-mode tools [merged]
#1788 - fix(responses): fail closed when a routed provider calls an undeclared tool [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[trungtaottn]

Observation: on the uncovered custom-Responses class, a clean [DONE] completion is also mis-accounted as a routing-health failure

This is a follow-on in support of #1809. The gap #1809 describes (a custom openai-responses
provider that completes output items but ends a turn without a Responses terminal —
response.completed / response.failed / response.incomplete — i.e. closes on a bare
data: [DONE] sentinel) affects not just client commit behavior but routing-health accounting.
On current dev that class is also recorded as an incomplete failure even though the stream
contents were a clean success.

Concrete, traced end-to-end against main @ v2.22.0:

• src/server/relay.ts:1121-1126 — consumeForInspection’s onCleanEof fires
  onTerminal("incomplete") whenever the SSE inspector hasn’t reported a terminal ([DONE] is not a
  terminal — terminalStatusFromParsed(null) for [DONE] at relay.ts:280). So a [DONE]-only
  stream is accounted incomplete, never completed.
• src/server/responses/core.ts:2797-2816 — that synthetic terminal calls terminalRecorder("incomplete")
  • options.onNativePassthroughTerminal("incomplete").
• src/server/index.ts:1220-1224 (HTTP namespace) — onNativePassthroughTerminal →
  addFinalRequestLog(..., { terminalStatus: "incomplete", closeReason: "terminal" }).
• src/server/request-log.ts:319-332 — persisted to usage.jsonl with terminalStatus: "incomplete"
  (kept because it is not "completed").
• src/routing/history/indexer.ts:176 — requests.terminal_status = "incomplete" in the routing
  index.
• src/routing/health.ts:210 — classifySample: terminalStatus === "incomplete" ⇒ "failure"
  (and :304 increments incompleteStreams, and :212 already treats any terminal ≠ completed as
  failure). The health score is reduced both by the failure count and by the
  INCOMPLETE_WEIGHT * (1 - incompleteRate) term (health.ts:392-395).
• This surfaces in the documented routing-analytics incomplete-stream rate (src/routing/analytics.ts:356;
  docs-site/.../routing.md:229).

Reachable class (and why it’s not a standalone issue)

The step above only runs when the upstream has no terminal repair, and current repair is strictly
registry-controlled:

• providerModelResponsesTerminalRepair (src/providers/registry.ts:2635) resolves against the
  registry entry (:2641); a policy exists only for deepseek-v4-flash / deepseek-v4-pro
  (registry.ts:1519).
• core.ts:2642-2650 wraps the upstream body in relayResponsesSseWithTerminalRepair only when that
  policy is non-empty; a custom provider passes its body through unrepaired.
• Built-in DeepSeek V4 is repaired before the relays → never hits this. Canonical OpenAI forward-pool
  turns also don’t health-penalize: codexForwardTerminalOutcomeRecorder records an incomplete as a
  200 success (core.ts:654-667) — though the request-log/requests.terminal_status would still read
  incomplete.

So the mis-accounting is reachable only for a custom, unrepaired openai-responses provider that
closes a structurally-complete turn on a bare [DONE] — exactly the class this issue says is still
uncovered. I’m not filing it separately; it belongs here as implementation context, because the repair
escape hatch #1809 proposes is the natural place to also normalize the terminal-status accounting.

Ask for #1809

When adding the custom escape hatch, please require the repaired terminal to be recorded as
completed (not incomplete/failure) when the repaired stream is structurally complete, so a healthy
custom provider is not penalized in routing health / analytics on what is, from the client’s
perspective, a successful turn. (For contrast, the repair relay already emits the terminal into the
client body — responses-terminal-repair.ts:272-273 — whereas the passthrough relays emit nothing to
the client, only to accounting.)

Not a client-visible bug

To be explicit so nobody mis-files this: no response.incomplete SSE frame is injected into the
client body on the passthrough / eager (relay-eager.ts) / default tee paths. The raw [DONE] still
reaches and closes the client stream. The only contamination is the routing-health / request-log /
analytics status. (This corrects the earlier standalone draft’s headline.)

---