Skip to content

serve — /v1/models returns model ID with precision suffix, breaking OpenAI-compatible clients #1242

Description

@Novatoris

Describe the bug
/v1/models endpoint returns model ID with precision suffix (e.g. qualcomm/Qwen3-4B-Instruct-2507:W4A16), but using this exact ID in /v1/chat/completions fails with SDKError(Invalid input parameters or handle), quantization 'W4A16' not found for model 'qualcomm/Qwen3-4B-Instruct-2507'. The request only succeeds when the suffix is omitted. This breaks any OpenAI-compatible client that auto-populates the model ID from /v1/models (confirmed with AnythingLLM).

To Reproduce

  1. Pull a QAIRT model: geniex pull ai-hub-models/Qwen3-4B-Instruct-2507
  2. Start server: geniex serve --host 127.0.0.1:18181 --compute npu
  3. Query /v1/models:
Invoke-RestMethod -Uri "http://127.0.0.1:18181/v1/models" -Method Get

Returns qualcomm/Qwen3-4B-Instruct-2507:W4A16

  1. Use the returned ID in a chat request:
Invoke-RestMethod -Uri "http://127.0.0.1:18181/v1/chat/completions" -Method Post -ContentType "application/json" -Body '{"model": "qualcomm/Qwen3-4B-Instruct-2507:W4A16", "messages": [{"role": "user", "content": "Say hello"}]}'
  1. See error: SDKError(Invalid input parameters or handle), quantization 'W4A16' not found

Expected behavior
/v1/models should return a model ID that works directly with /v1/chat/completions. Either the precision suffix should be stripped from the models list, or the completions endpoint should accept it.

Workaround
Clear the Precision field in geniex.json:

$json = Get-Content "C:\LocalAI\geniex\models\models\qualcomm\Qwen3-4B-Instruct-2507\geniex.json" | ConvertFrom-Json
$json.Precision = ""
$json | ConvertTo-Json -Depth 10 | Set-Content "C:\LocalAI\geniex\models\models\qualcomm\Qwen3-4B-Instruct-2507\geniex.json"

Desktop:

  • OS: Windows 11 ARM64
  • Device: Snapdragon X Plus X1P42100
  • GenieX CLI Version: v0.3.17
  • QAIRT Runtime Version: v2.45.0.260326
  • LlamaCPP Runtime Hash: ae9291e
  • Model: ai-hub-models/Qwen3-4B-Instruct-2507 (W4A16, QAIRT runtime)
  • AnythingLLM Version: v1.15.0 (used for OpenAI-compatible client testing)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions