fix: keep gateway model ids intact for OpenAI-compatible embeddings - #1843
Open
asiqur-rahman wants to merge 1 commit into
Open
fix: keep gateway model ids intact for OpenAI-compatible embeddings#1843asiqur-rahman wants to merge 1 commit into
asiqur-rahman wants to merge 1 commit into
Conversation
LiteLLMEmbeddingWrapper dropped the provider prefix whenever the resolved
provider was openai, so the raw model id reached litellm.embedding() and
LiteLLM re-parsed its first path segment as a provider. Gateway model ids
that contain a slash were therefore mangled or rejected before the request
left the process:
nvidia/llama-nemotron-embed-vl-1b-v2:free -> LLM Provider NOT provided
auto/embedding -> LLM Provider NOT provided
openrouter/openai/text-embedding-3-small -> routed as provider openrouter,
forwarded as openai/text-embedding-3-small
Always prefix instead, matching LiteLLMChatWrapper, which is why chat models
already work against the same endpoints. LiteLLM strips a recognised openai/
prefix and forwards the remainder verbatim, so plain OpenAI model ids resolve
exactly as before.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1597.
Problem
LiteLLMEmbeddingWrapper.__init__drops the provider prefix whenever the resolved provider isopenai:Five embedding providers in
conf/model_providers.yamlresolve tolitellm_provider: openai—openai,openrouter,a0_venice,venice, andother("Other OpenAI compatible"). For all ofthem the model id reaches
litellm.embedding()with no provider attached, so LiteLLM re-parses itsfirst path segment as a provider name. Any id containing a
/is then either rejected or silentlyrewritten before the request leaves the process.
This fails inside
Memory.initialize()→embedder.embed_query("example"), so the agent cannotbuild its FAISS index at all.
Before / after
Produced by calling
models.get_embedding_model(...)and feeding the resultingmodel_nametolitellm.get_llm_provider()(litellm 1.88.1):openrouteropenai/text-embedding-3-smalltext-embedding-3-small—openai/silently eatenopenai/text-embedding-3-smallopenrouternvidia/llama-nemotron-embed-vl-1b-v2:freeBadRequestError: LLM Provider NOT providednvidia/llama-nemotron-embed-vl-1b-v2:freeotheropenrouter/openai/text-embedding-3-smallopenrouter, sentopenai/text-embedding-3-smallopenrouter/openai/text-embedding-3-smallothernvidia/llama-nemotron-embed-vl-1b-v2:freeBadRequestError: LLM Provider NOT providednvidia/llama-nemotron-embed-vl-1b-v2:freeopenaitext-embedding-3-smalltext-embedding-3-smalltext-embedding-3-small(unchanged)ollamanomic-embed-textnomic-embed-textnomic-embed-text(unchanged)Row 1 is worth calling out: for the bundled OpenRouter provider the id is not rejected, it is
quietly truncated, so OpenRouter receives a model it does not recognise.
Row 3 explains an error that looks like a gateway fault but is not — a gateway replying
No credentials for embedding provider: openai. LiteLLM had strippedopenrouter/and forwardedopenai/text-embedding-3-small, so the gateway tried to resolveopenaias one of its upstreams.It was answering a mangled request correctly.
Fix
Always prefix, which is exactly what
LiteLLMChatWrapper.__init__(models.py:383) already does:That asymmetry is why chat models already work against these endpoints while embeddings do not,
from the same provider config and the same
api_base. LiteLLM strips a recognisedopenai/prefixand forwards the remainder verbatim, so plain OpenAI ids resolve exactly as before (row 5), and
providers that do not resolve to
openaiwere already taking the prefixing branch and are untouched(row 6).
No provider-specific branching, no new dependency, and LiteLLM stays on the path for every provider.
Verification
Live gateway (OpenAI-compatible gateway,
Other OpenAI compatible+ custom base URL):openrouter/openai/text-embedding-3-small— before:BadRequestError ... No credentials for embedding provider: openai; after: HTTP 200, 1536-dimension embedding returned.nvidia/llama-nemotron-embed-vl-1b-v2:free,auto/embedding) still fail after thechange, but with gateway-side model errors (
404 page not found,Unknown embedding provider: auto) rather than LiteLLM provider-resolution errors. Confirmed withcurlstraight at thegateway that those two ids fail identically without Agent Zero in the path, so they are gateway
configuration, not this bug. The change is what allows the gateway to answer for itself.
Tests:
pytest tests/test_model_config_api_keys.py— 24 passed.Whole suite, run in a minimal local environment and compared against unmodified
upstream/readyin a second worktree:
upstream/readyIdentical pre-existing failures and collection errors on both sides, with exactly one additional
pass on this branch (the new test). Those failures and errors come from optional dependencies
missing in my environment (
giturlparse,aiogram,langchain_community,starlette,soundfile,fastmcp) and are unrelated to this change — no traceback touchesmodels.py.Happy to re-run anything specific in a full environment if useful.
Added
test_openai_compatible_embedding_keeps_gateway_model_string, following the existing style ofthat file (asserting
model_nameandkwargs["api_base"]). It covers the gateway id shapes forother, both OpenRouter id shapes, the plain-OpenAI no-regression case, andollamaas anunaffected control.
Scope
LiteLLMEmbeddingWrapper.model_namehas no consumers outsidemodels.py— it is only ever passedas
model=tolitellm.embedding(). In particular, memory-index validity is keyed onmodel_config.provider/model_config.name, the configured values(
plugins/_memory/helpers/memory.py:202and:242), not on the wrapper'smodel_name. So thischange does not invalidate any existing FAISS index or trigger a reindex.
Providers whose
litellm_provideris notopenaialready took the prefixing branch and arebit-for-bit unchanged (verified for
ollamaandlm_studio).One deliberate behaviour change
A model id that already carries the
openai/prefix under theopenaiprovider now keeps it:openaitext-embedding-3-smalltext-embedding-3-smalltext-embedding-3-smallopenaiopenai/text-embedding-3-smalltext-embedding-3-smallopenai/text-embedding-3-smallAnyone who typed the redundant prefix into the OpenAI provider should drop it and configure just
text-embedding-3-small. This matches howLiteLLMChatWrapperhas always behaved for chat models.It is worth spelling out why a "skip the prefix if the model already starts with it" guard is not
the answer here: the bundled OpenRouter provider legitimately uses ids of exactly that shape, e.g.
openai/text-embedding-3-small, and at this point in the code the two cases are indistinguishable —both arrive as provider
openaiwith a model id beginningopenai/. Such a guard would restore theOpenAI edge case at the cost of re-breaking OpenRouter, which is the bug this PR fixes. The two are
told apart only by
api_base, which belongs to the provider config rather than the model id.Alternative considered
Passing
custom_llm_provider="openai"tolitellm.embedding()and leaving the model id untouchedalso works — I verified both against a live gateway and both return a correct 1536-dimension
embedding for
openrouter/openai/text-embedding-3-small.I went with the prefix because it is a one-line change that makes the embedding wrapper match the
chat wrapper, so there is one rule in the file instead of two. It does not avoid the edge case above
either — a redundant
openai/would then be forwarded verbatim instead of prefixed, which is wrongfor the OpenAI provider in the same way. Happy to switch to
custom_llm_providerif you wouldrather keep model ids textually untouched; say the word and I will push that version instead.