Skip to content

fix: keep gateway model ids intact for OpenAI-compatible embeddings - #1843

Open
asiqur-rahman wants to merge 1 commit into
agent0ai:readyfrom
asiqur-rahman:fix-openai-compatible-embedding-model-prefix
Open

fix: keep gateway model ids intact for OpenAI-compatible embeddings#1843
asiqur-rahman wants to merge 1 commit into
agent0ai:readyfrom
asiqur-rahman:fix-openai-compatible-embedding-model-prefix

Conversation

@asiqur-rahman

Copy link
Copy Markdown

Fixes #1597.

Problem

LiteLLMEmbeddingWrapper.__init__ drops the provider prefix whenever the resolved provider is openai:

self.model_name = f"{provider}/{model}" if provider != "openai" else model

Five embedding providers in conf/model_providers.yaml resolve to litellm_provider: openai
openai, openrouter, a0_venice, venice, and other ("Other OpenAI compatible"). For all of
them the model id reaches litellm.embedding() with no provider attached, so LiteLLM re-parses its
first path segment as a provider name. Any id containing a / is then either rejected or silently
rewritten before the request leaves the process.

This fails inside Memory.initialize()embedder.embed_query("example"), so the agent cannot
build its FAISS index at all.

Before / after

Produced by calling models.get_embedding_model(...) and feeding the resulting model_name to
litellm.get_llm_provider() (litellm 1.88.1):

provider configured model before → sent to endpoint after → sent to endpoint
openrouter openai/text-embedding-3-small text-embedding-3-smallopenai/ silently eaten openai/text-embedding-3-small
openrouter nvidia/llama-nemotron-embed-vl-1b-v2:free BadRequestError: LLM Provider NOT provided nvidia/llama-nemotron-embed-vl-1b-v2:free
other openrouter/openai/text-embedding-3-small routed as provider openrouter, sent openai/text-embedding-3-small openrouter/openai/text-embedding-3-small
other nvidia/llama-nemotron-embed-vl-1b-v2:free BadRequestError: LLM Provider NOT provided nvidia/llama-nemotron-embed-vl-1b-v2:free
openai text-embedding-3-small text-embedding-3-small text-embedding-3-small (unchanged)
ollama nomic-embed-text nomic-embed-text nomic-embed-text (unchanged)

Row 1 is worth calling out: for the bundled OpenRouter provider the id is not rejected, it is
quietly truncated, so OpenRouter receives a model it does not recognise.

Row 3 explains an error that looks like a gateway fault but is not — a gateway replying
No credentials for embedding provider: openai. LiteLLM had stripped openrouter/ and forwarded
openai/text-embedding-3-small, so the gateway tried to resolve openai as one of its upstreams.
It was answering a mangled request correctly.

Fix

Always prefix, which is exactly what LiteLLMChatWrapper.__init__ (models.py:383) already does:

model_value = f"{provider}/{model}"

That asymmetry is why chat models already work against these endpoints while embeddings do not,
from the same provider config and the same api_base. LiteLLM strips a recognised openai/ prefix
and forwards the remainder verbatim, so plain OpenAI ids resolve exactly as before (row 5), and
providers that do not resolve to openai were already taking the prefixing branch and are untouched
(row 6).

No provider-specific branching, no new dependency, and LiteLLM stays on the path for every provider.

Verification

Live gateway (OpenAI-compatible gateway, Other OpenAI compatible + custom base URL):

  • openrouter/openai/text-embedding-3-small — before: BadRequestError ... No credentials for embedding provider: openai; after: HTTP 200, 1536-dimension embedding returned.
  • Two other ids (nvidia/llama-nemotron-embed-vl-1b-v2:free, auto/embedding) still fail after the
    change, but with gateway-side model errors (404 page not found, Unknown embedding provider: auto) rather than LiteLLM provider-resolution errors. Confirmed with curl straight at the
    gateway that those two ids fail identically without Agent Zero in the path, so they are gateway
    configuration, not this bug. The change is what allows the gateway to answer for itself.

Tests: pytest tests/test_model_config_api_keys.py — 24 passed.

Whole suite, run in a minimal local environment and compared against unmodified upstream/ready
in a second worktree:

failed passed collection errors
upstream/ready 105 972 33
this branch 105 973 33

Identical pre-existing failures and collection errors on both sides, with exactly one additional
pass on this branch (the new test). Those failures and errors come from optional dependencies
missing in my environment (giturlparse, aiogram, langchain_community, starlette,
soundfile, fastmcp) and are unrelated to this change — no traceback touches models.py.
Happy to re-run anything specific in a full environment if useful.

Added test_openai_compatible_embedding_keeps_gateway_model_string, following the existing style of
that file (asserting model_name and kwargs["api_base"]). It covers the gateway id shapes for
other, both OpenRouter id shapes, the plain-OpenAI no-regression case, and ollama as an
unaffected control.

Scope

LiteLLMEmbeddingWrapper.model_name has no consumers outside models.py — it is only ever passed
as model= to litellm.embedding(). In particular, memory-index validity is keyed on
model_config.provider / model_config.name, the configured values
(plugins/_memory/helpers/memory.py:202 and :242), not on the wrapper's model_name. So this
change does not invalidate any existing FAISS index or trigger a reindex.

Providers whose litellm_provider is not openai already took the prefixing branch and are
bit-for-bit unchanged (verified for ollama and lm_studio).

One deliberate behaviour change

A model id that already carries the openai/ prefix under the openai provider now keeps it:

provider configured model before → sent after → sent
openai text-embedding-3-small text-embedding-3-small text-embedding-3-small
openai openai/text-embedding-3-small text-embedding-3-small openai/text-embedding-3-small

Anyone who typed the redundant prefix into the OpenAI provider should drop it and configure just
text-embedding-3-small. This matches how LiteLLMChatWrapper has always behaved for chat models.

It is worth spelling out why a "skip the prefix if the model already starts with it" guard is not
the answer here: the bundled OpenRouter provider legitimately uses ids of exactly that shape, e.g.
openai/text-embedding-3-small, and at this point in the code the two cases are indistinguishable —
both arrive as provider openai with a model id beginning openai/. Such a guard would restore the
OpenAI edge case at the cost of re-breaking OpenRouter, which is the bug this PR fixes. The two are
told apart only by api_base, which belongs to the provider config rather than the model id.

Alternative considered

Passing custom_llm_provider="openai" to litellm.embedding() and leaving the model id untouched
also works — I verified both against a live gateway and both return a correct 1536-dimension
embedding for openrouter/openai/text-embedding-3-small.

I went with the prefix because it is a one-line change that makes the embedding wrapper match the
chat wrapper, so there is one rule in the file instead of two. It does not avoid the edge case above
either — a redundant openai/ would then be forwarded verbatim instead of prefixed, which is wrong
for the OpenAI provider in the same way. Happy to switch to custom_llm_provider if you would
rather keep model ids textually untouched; say the word and I will push that version instead.

LiteLLMEmbeddingWrapper dropped the provider prefix whenever the resolved
provider was openai, so the raw model id reached litellm.embedding() and
LiteLLM re-parsed its first path segment as a provider. Gateway model ids
that contain a slash were therefore mangled or rejected before the request
left the process:

  nvidia/llama-nemotron-embed-vl-1b-v2:free  -> LLM Provider NOT provided
  auto/embedding                            -> LLM Provider NOT provided
  openrouter/openai/text-embedding-3-small  -> routed as provider openrouter,
                                               forwarded as openai/text-embedding-3-small

Always prefix instead, matching LiteLLMChatWrapper, which is why chat models
already work against the same endpoints. LiteLLM strips a recognised openai/
prefix and forwards the remainder verbatim, so plain OpenAI model ids resolve
exactly as before.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant