Skip to content

serve — Streaming response always returns empty finish_reason instead of null/"stop", breaking OpenAI-compatible clients #1243

Description

@Novatoris

Describe the bug
When using geniex serve with streaming enabled ("stream": true), all chunks in the streaming response have "finish_reason": "" (empty string) including the final chunk. According to the OpenAI streaming specification, finish_reason should be null on intermediate chunks and "stop" on the final chunk. This causes OpenAI-compatible clients such as AnythingLLM to never detect that generation has finished, requiring the user to manually stop generation.

Non-streaming responses ("stream": false) correctly return "finish_reason": "stop".

To Reproduce

  1. Start server: geniex serve --host 127.0.0.1:18181 --compute npu
  2. Send a streaming request:
Invoke-WebRequest -Uri "http://127.0.0.1:18181/v1/chat/completions" -Method Post -ContentType "application/json" -UseBasicParsing -Body '{"model": "qualcomm/Qwen3-4B-Instruct-2507", "messages": [{"role": "user", "content": "Say hello"}], "stream": true}' | Select-Object -ExpandProperty Content
  1. Observe all chunks have "finish_reason": "" including the final chunk before data: [DONE]

Expected behavior
Per the OpenAI streaming specification:

  • Intermediate chunks: "finish_reason": null
  • Final chunk: "finish_reason": "stop" with empty "delta": {}

GenieX currently sends "finish_reason": "" (empty string) on all chunks including the final one, which is non-compliant and causes clients to never detect completion.

Workaround
Manually stop generation in the UI once the response appears complete. In AnythingLLM, click the stop button after the model finishes generating text — the response is complete but the client keeps waiting due to the missing finish_reason: stop signal.

Desktop:

  • OS: Windows 11 ARM64
  • Device: Snapdragon X Plus X1P42100
  • GenieX CLI Version: v0.3.17
  • QAIRT Runtime Version: v2.45.0.260326
  • LlamaCPP Runtime Hash: ae9291e
  • Model: ai-hub-models/Qwen3-4B-Instruct-2507 (W4A16, QAIRT runtime)
  • AnythingLLM Version: v1.15.0 (used for OpenAI-compatible client testing)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions