Skip to content

Cosmos3-edge PR 5: end a failed chat stream with an error event instead of stop - #296

Open
merceod wants to merge 2 commits into
mainfrom
engine/chat-stream-errors
Open

merceod wants to merge 2 commits into
mainfrom
engine/chat-stream-errors

Conversation

@merceod

@merceod merceod commented Sep 22, 2026

Copy link
Copy Markdown
Collaborator

APIServer.iter_result_chunks already reports a request that fails after its stream opened (an engine error mid generation, a delivery timeout) as a terminal error chunk, and /generate and /v1/videos forward it. The /v1/chat/completions SSE path dropped it and closed with finish_reason: "stop", so a truncated answer read as a complete one. I ran into this while benchmarking concurrent Cosmos3-Edge reasoner requests, where worker errors showed up as short answers with a normal stop and the throughput numbers counted them.

The stream now ends with an OpenAI-style error event and [DONE], no stop chunk, matching the router's non-streaming error body.

data: {"error": {"message": "Error in worker: ...", "type": "server_error", "code": 500}}

data: [DONE]

Test in test/modular/test_openai_router.py::test_chat_stream_reports_a_failed_request_in_band (stubbed API). ruff and test/modular pass.

@merceod
merceod force-pushed the engine/chat-stream-errors branch 2 times, most recently from 0f6dfa0 to 105bc2a Compare September 23, 2026 23:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant