Repository navigation
feat(services): integrate LLMLingua compression and Bifrost routing - #5
Merged
Merged
Conversation
Port cheaper-inference#12967 and cheaper-inference#12953 onto dev. Run Bifrost inside the native executor pipeline to preserve authentication, policy, guardrails and accounting, with cancellation and native fallback. Replace the placeholder compression server with the existing ONNX worker and expose the LLMLingua service lifecycle through the dashboard. Co-authored-by: Rouzbeh <78313022+rqzbeh@users.noreply.github.com>
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
✨ Simplify code
Warning Some tools did not complete. Review the errors below. 🔧 Vale (3.17.1){ 🔧 ESLint
ESLint install timed out. The project may have too many dependencies for the sandbox. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Reviewer's GuidePorts LLMLingua HTTP compression and Bifrost ingress while adapting both features to OmniRoute’s existing service supervisor, ONNX worker, native executor, authentication/policy pipeline, response translation, cooldown, and header-safety mechanisms. LLMLingua remains opt-in and disabled by default; Bifrost is enabled explicitly and falls back to native execution on eligible failures, with focused regression and integration coverage. Sequence diagram for Bifrost ingress with native fallbacksequenceDiagram
participant Client
participant ChatPipeline
participant NativeExecutor
participant BifrostExecutor
participant Bifrost
Client->>ChatPipeline: handleChatCore()
ChatPipeline->>ChatPipeline: authenticate and apply policy
ChatPipeline->>NativeExecutor: resolveExecutorWithProxy()
NativeExecutor->>BifrostExecutor: wrapExecutorWithBifrost()
BifrostExecutor->>BifrostExecutor: shouldTryBifrostForRequest()
alt eligible OpenAI or Claude request
BifrostExecutor->>Bifrost: dispatchToBifrost()
alt successful response
Bifrost-->>BifrostExecutor: translated response
BifrostExecutor-->>ChatPipeline: transport=bifrost
else network or HTTP 5xx failure
BifrostExecutor->>BifrostExecutor: recordBifrostFailure()
BifrostExecutor->>NativeExecutor: execute()
NativeExecutor-->>BifrostExecutor: native response
BifrostExecutor-->>ChatPipeline: fallback response
end
else disabled or ineligible request
BifrostExecutor->>NativeExecutor: execute()
NativeExecutor-->>BifrostExecutor: native response
BifrostExecutor-->>ChatPipeline: native response
end
ChatPipeline-->>Client: response with safe routing headers
Sequence diagram for LLMLingua HTTP compression fallbacksequenceDiagram
participant CompressionEngine
participant LLMLinguaHTTP
participant ONNXWorker
CompressionEngine->>CompressionEngine: resolveBackend()
alt LLMLINGUA_BASE_URL configured
CompressionEngine->>LLMLinguaHTTP: POST /compress
LLMLinguaHTTP->>LLMLinguaHTTP: validate and bound request
LLMLinguaHTTP->>ONNXWorker: inference
alt valid compressed output
ONNXWorker-->>LLMLinguaHTTP: compressed text
LLMLinguaHTTP-->>CompressionEngine: compressed text and ratio
else unavailable or invalid result
LLMLinguaHTTP-->>CompressionEngine: HTTP error
CompressionEngine->>ONNXWorker: workerBackend()
ONNXWorker-->>CompressionEngine: native worker result
end
else base URL unset
CompressionEngine->>ONNXWorker: workerBackend()
ONNXWorker-->>CompressionEngine: native worker result
end
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
Raudbjorn
added a commit
that referenced
this pull request
Sep 7, 2026
* fix(oauth): Kiro import token endpoint no longer overwrites existing connection when using shared cached OIDC clientId (cheaper-inference#9435) Closes cheaper-inference#9435 (cherry picked from commit 9edefd4) * fix(translator): add case-insensitive fallback for upstream tool call name lookups (cheaper-inference#9575) Closes cheaper-inference#9575 (cherry picked from commit c9a3361) * fix(translator): restore original tool name casing in Gemini response translators (cheaper-inference#9568) Closes cheaper-inference#9568 (cherry picked from commit c9debe9) * fix(model): add aq alias for amazon-q provider so parseModel resolves it instead of falling back to OpenAI (cheaper-inference#9550) Closes cheaper-inference#9550 (cherry picked from commit f338363) * fix(translator): reject nameless streaming tool calls (cherry picked from commit e274819) * fix(devin-desktop): sanitize Claude Code prompt triggers and enforce tool budget (cherry picked from commit 8c1514a) (cherry picked from commit a18771b) * fix(devin-desktop): sanitize interactive Claude Code CLI identity declarations (cherry picked from commit e5553c1) (cherry picked from commit db089c8) * fix(claude-code): format context overflow as 'Prompt is too long' to trigger reactive compact (cherry picked from commit b5fec6b) (cherry picked from commit 5e83dc6) * fix(chat): import getCachedSettings from readCache (cherry picked from commit 999c708) (cherry picked from commit d8329dd) * fix(image): add 'agy' alias for antigravity image generation models (cherry picked from commit 2ac3e0d) (cherry picked from commit 203cdb3) * fix(translator): subtract cachedContentTokenCount from input_tokens in gemini-to-claude translator to avoid double-counting in Claude Code (cherry picked from commit 4c5ec77) (cherry picked from commit d82672d) * fix(translator): add cache_creation_input_tokens, update tests, and set auto-compression defaults (cherry picked from commit 9b68a4b) (cherry picked from commit 6f4e1f4) * fix(translator,compression): sanitize token metrics with Number.isFinite and restore opt-in compression defaults (cherry picked from commit 985c3c2) (cherry picked from commit cf9da8e) * fix(vision-bridge): cross-check agy/antigravity aliases and declare SWE vision capabilities (cherry picked from commit d1f2f5a) (cherry picked from commit a80b6df) * fix(vision-bridge): return skip if mixed combo has vision capable target (cherry picked from commit a049fbb) (cherry picked from commit fd2d093) * fix(ratelimit): set 120s maxWaitMs floor for long-context providers to prevent 504 on 100k+ tokens (cherry picked from commit 98cc4b0) (cherry picked from commit 6195ebf) * fix(ratelimit): clean up redundant zai-web entry, add null safety and long-context override tests (cherry picked from commit 743a118) (cherry picked from commit 4421ceb) * test(ratelimit): add direct assertion for opencode in queue wait floor test (cherry picked from commit dda584d) (cherry picked from commit 5b987d9) * fix(translator): synthesize fallback text block from reasoning when model emits only thinking for compaction (cherry picked from commit 0e13723) (cherry picked from commit bc9e1c5) * fix(devin-desktop): classify content policy trailer errors as deterministic 400 and bypass cooldown (cherry picked from commit 8820f15) (cherry picked from commit f8a4a3b) * feat(pricing): add SWE-1.7 Lightning pricing for devin-desktop and windsurf providers (cherry picked from commit 11ec472) (cherry picked from commit 0956840) * feat(pricing): set swe-1-7 price at half Lightning rate (.25/zsh.5/.25 per 1M) (cherry picked from commit 60def8f) (cherry picked from commit 526c814) * feat(models): add Gemini 3.8 Flash tiers and pricing Introduce Gemini 3.8 Flash across OmniRoute on top of the existing 3.6/3.7 Flash support: - modelSpecs: gemini-3.8-flash{,-high,-medium,-low,-tiered} with 1M context, 64K output, defaultReasoningEffort=medium. - pricing: promo $0.75/$3.75 per 1M for gemini, antigravity/agy, frontier-labs, cheaperinference, orcarouter, and inference-hosts. - provider registries: gemini, gemini-web, github, ghe-copilot, cursor, cheaperinference, kilocode, opencode/zen, orcarouter, raycast, tinycms, lmarena. - AGY/Antigravity catalogs, aliases and free-model catalog. - Fallback services: conol, githubCopilot, notionWeb, promptql. - CLI tool model aliases and modelAliasSeed. - modelFamilyFallback chains for the 3.8 Flash family. - geminiRateLimits.json. Tests: t28, models-catalog-route, model-capabilities-registry, pricing-ag-flash-tiers, agy-gemini-3696-tier-passthrough and model-family-fallback-notation all pass. npm run typecheck:core and npm run lint pass. (cherry picked from commit 7e44a6d) (cherry picked from commit c8391ff) * fix(chat): return resource pressure guard as a normal result and avoid cooldown executeChatWithBreaker previously returned { localResourcePressureResult } when the resource pressure guard tripped, but the caller in chat.ts only destructures { result, tlsFingerprintUsed } and immediately reads result.success. That caused a TypeError ('Cannot read properties of undefined (reading "success")') and made combo fallback logs show a confusing crash. Now the pressure guard is returned as a normal { result } with errorCode/errorType "resource_pressure", and chat.ts surfaces the 503 immediately without marking the connection unavailable. (cherry picked from commit 22b641e) (cherry picked from commit ee2af6f) * ops(docker): raise runtime heap to 4096MB and pin tls-client native version OMNIROUTE_MEMORY_MB 1024MB was too tight for production traffic with large fusion-combo panels, causing constant GC pressure and PSI CPU/memory critical states. Raise the default runtime heap ceiling to 4096MB. Also pin TLS_CLIENT_VERSION=1.15.1 so the tls-client-node postinstall does not fetch an upstream "latest" release that may be missing the Linux .so asset. (cherry picked from commit db021d9) (cherry picked from commit 8583cfe) * fix(providers): harden empty responses and oauth test classification - stream: inject a space instead of empty text for the synthetic Claude empty-response fallback so clients do not trip on zero-length completions (502 noise seen in production call_logs). - providers/test: return statusCode 400 for unsupported provider tests; add custom-oauth-connection-proxy to OAUTH_TEST_CONFIG (checkExpiry only — the proxy has no userinfo endpoint). - resolveRoutingModel: default subagent routing model claude-sonnet-5 -> claude-haiku-4.5 to reduce pressure on the antigravity lane. (cherry picked from commit c219b43) (cherry picked from commit 2892e9d) * feat: port subagent routing + tool-call shim + devin/transport hardening from local production (cherry picked from commit 2cb7b19) * test: fix gemini-to-claude usage test fixture, drop repairedRaw tests (cherry picked from commit 53462ff) * chore: prune unused eslint suppressions (cherry picked from commit 8515409) * fix(docker): relax healthcheck to tolerate event-loop stalls under load The 5s timeout / 3 retries / 15s start-period was too strict for this single-process app: /healthz probes timed out during startup module loading and under sustained heavy /v1/messages traffic (~180k-token requests), flipping the container unhealthy and causing Docker/Dokploy to kill mid-session — the observed restart loop / 502 source. start-period=120s covers cold start; timeout=15s + retries=5 tolerate transient event-loop stalls (~150s) while still catching dead processes. (cherry picked from commit c749903) * fix(devin-desktop): accumulate id-less tool-call argument deltas Devin Desktop streams tool-call arguments as separate proto messages carrying only field 3 (arguments) with no id/name. The decoder dropped them (`if (toolCall.id)`), so clients received tool_use blocks with empty input {} — Claude Code then rejected ~50% of tool calls with InputValidationError, and the empty calls in history taught the model to emit empty arguments itself, causing a self-reinforcing error loop. Keep argument-only deltas and append them to the most recently started tool call so input_json_delta reaches the client. (cherry picked from commit 8a82a13) * fix(translator): purge never-succeedable empty tool_use pairs from history Clients whose history was poisoned by lost tool-call arguments keep replaying tool_use {input:{}} + InputValidationError pairs, teaching the model to emit empty arguments on new calls (self-reinforcing loop). When translating a Claude-format request, drop tool_use blocks whose input is empty AND whose tool schema declares required params (such a call could never have succeeded), along with their matching tool_result blocks and "(empty response)" placeholder texts. Tools with no required params (e.g. chrome_read_page) and tools absent from the tools list are left untouched. (cherry picked from commit 164c61d) * fix(review): adversarial-review hardening + deterministic stream error status Code-review patches: - chatDispatch: detect resource_pressure via typed error fields (dead localResourcePressureResult check evicted session affinity wrongly) - chat: return resource-pressure 503 before safeLogEvents so it is not logged as an upstream error; fix stale "retrying once" log message - stop.mjs: share isTestEnvironment() guard across POSIX and Win32 kill paths - supervisor: mkdir log dir before createWriteStream (ENOENT crash) - resolveRoutingModel: trim SUBAGENT_DEFAULT_MODEL env (whitespace-safe) - claudeHelper: drop string-content "(empty response)" messages; normalize Claude Code file-unchanged sentinel for non-Anthropic Claude-shape targets - toolCallShim: AskUserQuestion option objects emit only schema fields, question arrays join cleanly, options-only calls get a fallback question - devin-desktop: sanitizeDevinPrompt scoped to system/developer roles only; document parallel tool-call arg-delta attribution limitation - Dockerfile: healthcheck start-period 120s -> 60s (faster dead-detection, stall tolerance kept via timeout/retries) Compact-incident fix (session log analysis): - streamReadiness: a stream ending with only a deterministic upstream diagnostic (content policy / safety filter) now surfaces its real 400 status instead of a retryable 502 STREAM_EARLY_EOF — stops the observed ~9x/10min retry loop on payloads that can never succeed - sameAccountTransportRetry: 403 only retryable for the Devin permission_denied-internal-error quirk; content-policy text never retries - test: bound assertion follows SAME_ACCOUNT_TRANSPORT_RETRY_MAX (cherry picked from commit e37ae2b) * fix(translator): error event for upstream-truncated tool-call arguments When the upstream stream ends mid-tool-call (observed: Devin Desktop Connect streams cut during argument deltas), the accumulated arguments buffer is unparseable JSON. Closing the tool_use block normally handed the client a truncated partial_json - Claude Code raised InputValidationError, and the model retried with hallucinated stub args (seen across recent session logs). The openai-to-claude finish path now validates each non-shimmed tool call's argBuffer and emits a terminal error event instead of closing the block, so the turn is retried rather than executed as a truncated command. Shimmed tools are exempt (their shim repairs the buffer at close). (cherry picked from commit 4dca2f3) * feat(translator): schema-driven tool-call argument repair Models that emit wrong-typed fields, extra keys, or missing required args hit client-side InputValidationError retry loops (observed across recent Claude Code session logs). Add a generic sanitizer that repairs parsed tool arguments against the client-declared input_schema: coerce unambiguous type mismatches (stringified numbers/booleans, scalar to array, JSON-stringified objects), drop undeclared keys on closed schemas, fill missing required fields with type-correct empties - nested and depth-capped. openai-to-claude now buffers arguments for any tool with a declared schema (previously only named-shim tools) and emits one corrected input_json_delta at block close. The finish path also validates every accumulated argBuffer and emits a terminal error event when upstream truncated the JSON mid-call, so the turn is retried instead of executing a cut command. chatCore now passes whichever body still carries tools into the stream state - executor paths that consume/re-key the tools array (e.g. Devin Desktop) left the dispatched body without client schemas. (cherry picked from commit 84754f5) * fix(devin-desktop): encode CompletionConfig to stop silent 1024-token output cap The Devin Desktop Connect GetChatMessage wire was missing the CompletionConfiguration sub-message (field cheaper-inference#8). Without it, the upstream server defaulted output to ~1024 tokens for swe-1-7, which truncated Claude Code multi-tool turns (e.g. five Agent calls), produced empty Agent payloads, and led to the mid-response server error. Encode field cheaper-inference#8 with the OpenAI request's max_tokens (default 8192), temperature clamped to >=0.001, top_p, and top_k. Set max_newlines to a safe context-window value so it cannot accidentally become the binding output cap. Log the resolved max_tokens for monitoring. Update the golden fixture and add a focused test for custom values and the temperature=0 clamp. (cherry picked from commit 62effb4) * feat: add Kiro PWA bridge and websocket proxy support Publish the Kiro OAuth/PWA bridge flow together with websocket proxy scripts and route updates so the branch can be shared for iteration before full test validation completes. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> (cherry picked from commit 056a413) * feat(kiro): implement automatic account deactivation on quota exhaustion (cherry picked from commit 6a98790) * fix(health): add public /api/health route for external liveness probes - Add '/api/health' to PUBLIC_READONLY_API_ROUTE_PREFIXES so the auth pipeline classifies it as public (GET-only). - Create src/app/api/health/route.ts with a lightweight { status, timestamp } response, parallel to the existing /api/health/ping handler. External healthcheck/monitoring services probing /api/health were returning HTTP 401 (AUTH_001) because the route was not in any public prefix list. /api/monitoring/health was public but /api/health was not. (cherry picked from commit 6dac784) * feat: implement Freebuff CLI emulator and provider meta-service for connection and quota management (cherry picked from commit 1ac3383) * feat(freebuff): flip default from opt-in to opt-out Previously the Freebuff provider required FREEBUFF_ENABLED=1 to be exposed under /providers/freebuff. This commit flips the default so the provider is on by default (no env var required) and FREEBUFF_ENABLED=0 disables it explicitly. Changes: - base.ts: isFreebuffEnabled() returns true when env var is unset, empty, or any value other than 0/false. - registry.ts: error messages reference FREEBUFF_ENABLED=0 instead of =1. - en.json / fr.json: notEnabled.description updated to match. - .env.example + docs/providers/freebuff*.md: documentation reflects the new default and the disable-by-setting-FREEBUFF_ENABLED=0 flow. - freebuff-base.test.ts + freebuff-registry.test.ts: tests for the new default behaviour. (cherry picked from commit 050a34c) * fix(security): resolve freebuff public client credential and satisfy db re-export gate (cherry picked from commit a5b141f) * fix(review): address PR #1 review threads - chore(changelog): drop four stale fragments already in CHANGELOG (cheaper-inference#9435, cheaper-inference#9550, cheaper-inference#9568, cheaper-inference#9575) — stale-fragment gate exits nonzero on duplicate - fix(translator/claudeHelper): re-filter empty messages after the Pass 1.4 nameless-tool_use drop so non-final assistant messages whose only content was a nameless block do not survive into the request (#5, #6) - fix(ws/responses-ws-proxy): serialize concurrent forwardClientMessage calls via dispatchQueue so a second client message arriving during ensureUpstream does not drop the late frame (cheaper-inference#7) - fix(ws/v1-ws-bridge): close the session with code 1002 after a frame decode failure so an undecodable byte run does not loop forever (cheaper-inference#8) - fix(ws/both): send the WebSocket close frame BEFORE flipping this.closed so the sendFrame guard does not turn the close handshake into a no-op (cheaper-inference#9) - fix(oauth/freebuff): add 'network_error' to freebuffPollResponseSchema so the network-error return path typechecks (cheaper-inference#10) - fix(oauth/freebuff): declare onProgress on FreebuffPollOptions so the poll-loop callback typechecks (cheaper-inference#11) - fix(freebuff/cliEmulator): add readonly ok to FreebuffHttpResponse and populate it in tlsClientFetch, wreqFetch, and globalFetch so successful responses are not misclassified as failures (cheaper-inference#12) - test(translator/openai-to-claude): defer content_block_start until a name arrives and throw on a permanently nameless streamed call so the Muse/GLM cheaper-inference#2077 sequence and the nameless-rejection test both pass (cheaper-inference#16) - test(9568): resolve the three merge-marker regions in favor of HEAD — the test reflects the post-cheaper-inference#10392 translator behavior (cheaper-inference#13, cheaper-inference#14, cheaper-inference#15) * fix(review): address PR #1 review round 2 - docs(freebuff-api): replace nonexistent provider-specific routes with the generic [provider]/{models,limits,chat,embeddings,images} surface so the doc no longer documents 404-bound URLs (#1) - fix(translator/claudeHelper): scope the '(empty response)' placeholder drop to placeholders whose adjacent message actually had a paired tool_use/tool_result that this pass dropped; previously every historical placeholder was removed whenever any declared tool had required fields, silently deleting unrelated content and reordering turns - fix(translator/openai-to-claude): suppress input_json_delta passthrough while content_block_start is pending; replay the accumulated argBuffer in one shot after the late name arrives so the cheaper-inference#2077 Muse sequence produces a single start event with the full argument history instead of orphan deltas (cheaper-inference#16 follow-up) - fix(guardrails/visionBridge): a mixed combo (some targets lack vision) now returns 'process' instead of 'skip' so images are described before fan-out; matches the contract test at tests/unit/guardrails/visionBridge-combo-reroute.test.ts:147 - refactor(freebuff/httpClient): remove the tls-client-node fallback — it is not a declared dependency and tests/unit/tls-client-wreq-residue.test.ts asserts it stays absent; only wreq-js + global fetch remain as backends - fix(publicApiRoutes): remove '/api/health' from the CORS-relaxed list — the file's own comment says it must stay only in PUBLIC_READONLY_API_ROUTES_EXACT to avoid widening CORS - feat(pwa-register): wire shouldEnablePwaRegistration and isOmniRouteServiceWorkerScript into PwaRegister; loopback hostnames now unregister only matching OmniRoute workers rather than every worker and every cache entry - fix(oauth/kiro): extend CREDITS_EXHAUSTED_SIGNALS with Kiro-specific patterns ('usage limit exceeded', 'ThrottlingException') so the auto-deactivate path detects Kiro quota exhaustion; anchored on Kiro-distinguishing language to avoid broadening generic 429 classification - test(oauth-kiro-poll): drop 'as any' on connections[0] in favor of the inferred Record<string, unknown> from getProviderConnections --------- Co-authored-by: Diego Rodrigues de Sa e Souza <diegosouza.pw@gmail.com> Co-authored-by: Thinh <thinh0704hcm@users.noreply.github.com> Co-authored-by: Vonic <phanquochoipt@gmail.com> Co-authored-by: Aminetwiti <93622642+Aminetwiti@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com> Co-authored-by: Moutia <114811455+Moutia-Ben-Yahia@users.noreply.github.com> Co-authored-by: amine <amine@TweeDev.localdomain> Co-authored-by: Mohamed Benyahia <benyahiamohamd@gmail.com> Co-authored-by: Raudbjorn <Raudbjorn@users.noreply.github.com>
Raudbjorn
marked this pull request as ready for review
September 8, 2026 00:59
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Sorry @Raudbjorn, you've used your own review budget of 250,000 diff characters for the last 7 days.
You can request another review in 4 days and 15 hours by commenting @sourcery-ai review. Upgrade to get a review now.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_5dd86fd1-a62b-426e-b10f-5e39a5810b9c) |
There was a problem hiding this comment.
🟡 Changes recommended
The Bifrost executor wrapper mutates response headers in-place, which can throw at runtime when the native executor returns an immutable fetch() Response.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Integrates two opt-in sidecar capabilities into OmniRoute’s routing/compression pipeline: (1) supervised LLMLingua HTTP compression backed by the existing ONNX worker, and (2) Bifrost ingress dispatch at the executor layer so auth/policy/admission remain enforced by the normal pipeline.
Changes:
- Add LLMLingua as a supervised embedded service (install/start/stop/restart/update/status + dashboard tab) and an optional
LLMLINGUA_BASE_URLHTTP backend with fail-open fallback to the existing worker. - Introduce Bifrost ingress routing via
BIFROST_INGRESS_ENABLED=1, implemented as a wrapper around native executors, with cooldown/fallback and sanitized response-header propagation. - Update documentation, OpenAPI, env examples, and add focused unit/integration tests for the new behaviors.
Validation (custom)
- Commands run: none (review-only; no local execution in this session)
- Changed/added test files:
tests/unit/upstream-response-headers-strip.test.tstests/unit/services/installers/llmlingua.test.tstests/unit/services/bifrost/bifrostRouting.test.tstests/unit/services/bifrost/bifrostExecutor.test.tstests/unit/services/bifrost/bifrostClient.test.tstests/unit/compression/llmlingua-failopen.test.tstests/unit/api/v1/messages-bifrost-fastpath.test.tstests/unit/api/v1/chat-completions-bifrost-fastpath.test.tstests/integration/services/llmlingua-http.test.ts
- Test results: not executed in this session (custom)
File summaries
| File | Description |
|---|---|
tests/unit/upstream-response-headers-strip.test.ts |
Adds coverage ensuring only whitelisted sidecar diagnostics headers propagate. |
tests/unit/services/installers/llmlingua.test.ts |
Verifies LLMLingua service seed state, install script creation, and spawn args. |
tests/unit/services/bifrost/bifrostRouting.test.ts |
Covers env config parsing, backend selection, cooldown lifecycle, and reason parsing. |
tests/unit/services/bifrost/bifrostExecutor.test.ts |
Validates executor wrapping, protocol path selection, cooldown fallback, and cancellation. |
tests/unit/services/bifrost/bifrostClient.test.ts |
Tests unary + streaming dispatch, usage finalization, and credential/header safety. |
tests/unit/compression/llmlingua-failopen.test.ts |
Ensures HTTP sidecar is opt-in and fail-open behavior preserves non-empty content. |
tests/unit/api/v1/messages-bifrost-fastpath.test.ts |
Regression test that sidecar cannot bypass API-key auth for /v1/messages. |
tests/unit/api/v1/chat-completions-bifrost-fastpath.test.ts |
Regression test that sidecar cannot bypass API-key auth for /v1/chat/completions. |
tests/integration/services/llmlingua-http.test.ts |
Integration test for the supervised HTTP server: validation, size limits, and real worker output. |
src/shared/services/bifrost/bifrostRouting.ts |
New shared routing config, eligibility decision logic, and cooldown tracking. |
src/shared/services/bifrost/bifrostExecutor.ts |
Wraps native executors to optionally dispatch via Bifrost post-auth/policy/admission. |
src/shared/services/bifrost/bifrostClient.ts |
Implements safe dispatch to Bifrost with timeouts, streaming finalization, and header stripping. |
src/lib/services/installers/llmlinguaServer.ts |
Adds the generated Node HTTP server source that fronts the existing LLMLingua worker. |
src/lib/services/installers/llmlingua.ts |
Adds installer/update + spawn-args wiring for the supervised LLMLingua service. |
src/lib/services/bootstrap.ts |
Registers LLMLingua with embedded services bootstrap and avoids model sync for it. |
src/lib/db/migrations/174_llmlingua_service.sql |
Seeds a disabled-by-default version_manager row for the LLMLingua service. |
src/app/api/v1/relay/chat/completions/routingBackend.ts |
Refactors relay routing backend helpers to re-export from shared bifrost module. |
src/app/api/v1/relay/chat/completions/bifrostCooldown.ts |
Refactors relay cooldown helpers to re-export from shared bifrost module. |
src/app/api/services/llmlingua/_lib.ts |
Adds shared route helpers to create/register an LLMLingua supervisor on-demand. |
src/app/api/services/llmlingua/install/route.ts |
Adds loopback-only install endpoint for LLMLingua service. |
src/app/api/services/llmlingua/start/route.ts |
Adds loopback-only start endpoint with “not installed” handling. |
src/app/api/services/llmlingua/stop/route.ts |
Adds loopback-only stop endpoint. |
src/app/api/services/llmlingua/restart/route.ts |
Adds loopback-only restart endpoint with “not installed” handling. |
src/app/api/services/llmlingua/update/route.ts |
Adds loopback-only update endpoint with optional stop/restart if previously running. |
src/app/api/services/llmlingua/status/route.ts |
Adds loopback-only status endpoint merging supervisor + DB + installer state. |
src/app/api/services/llmlingua/auto-start/route.ts |
Adds loopback-only setter for auto-start flag (validated JSON body). |
src/app/api/services/llmlingua/auto-restart-adopted/route.ts |
Adds loopback-only setter for auto-restart-adopted flag (validated JSON body). |
src/app/(dashboard)/dashboard/providers/services/tabs/LlmlinguaServiceTab.tsx |
Adds Services dashboard tab UI for LLMLingua. |
src/app/(dashboard)/dashboard/providers/services/page.tsx |
Registers LLMLingua in the Services dashboard tabs list and renders it. |
open-sse/utils/upstreamResponseHeaders.ts |
Adds whitelisted propagation of sidecar diagnostics headers. |
open-sse/services/compression/engines/llmlingua/index.ts |
Adds an HTTP sidecar backend path, opt-in via LLMLINGUA_BASE_URL, fail-open to worker. |
open-sse/handlers/chatCore/streamingResponseHeaders.ts |
Propagates allowed sidecar diagnostics into streaming responses. |
open-sse/handlers/chatCore/nonStreamingResponseHeaders.ts |
Propagates allowed sidecar diagnostics into non-streaming responses. |
open-sse/handlers/chatCore/executorProxy.ts |
Wraps native executors with the Bifrost ingress wrapper when enabled. |
open-sse/handlers/chatCore.ts |
Plumbs provider headers into response-header builder for diagnostics propagation. |
docs/reference/ENVIRONMENT.md |
Documents fork sidecar env vars for Bifrost ingress + LLMLingua service integration. |
docs/ops/RELEASE_CHECKLIST.md |
Adds fork sidecar smoke-check guidance for LLMLingua and Bifrost ingress. |
docs/openapi.yaml |
Documents new LLMLingua service management endpoints as loopback-only. |
docs/frameworks/EMBEDDED-SERVICES.md |
Updates embedded services documentation to include LLMLingua + ingress behavior notes. |
changelog.d/features/llmlingua-bifrost-fork.md |
Adds a changelog fragment entry describing this integration. |
.env.example |
Documents and adds example env vars for BIFROST_INGRESS_ENABLED and LLMLingua settings. |
Review details
- Files reviewed: 41/41 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Wrap native fallback responses with copied headers instead of mutating immutable fetch responses. Preserve the body stream, status and executor metadata for both return shapes. Cover error and cooldown fallback with real immutable responses for streaming and nonstreaming requests. Remove the unused changelog fragment and retain upstream attribution in the embedded-services guide. Explain the worker callback concurrency guard. Review: #5 Base: Raudbjorn/OmniRoute feat/llmlingua-bifrost 793302f Validation: - New regression fails before the fix with TypeError: immutable. - 26 focused Bifrost, response-header and LLMLingua installer tests pass. - Focused ESLint, lint-staged --no-stash and check:any-budget:t11 pass. - typecheck:core reports duplicate provider identifiers/properties; reproduced on the original PR head. - check-docs-sync.mjs fails on removed CHANGELOG.md; check-tracked-artifacts.mjs flags two existing docs/superpowers files. Both reproduce on the original PR head. - Pre-commit checks run manually with --no-stash; hook automation disabled for this commit because the unchanged baseline gates fail. - Full suite, build and live provider/model behavior were not exercised.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_94b381c2-8aec-4d55-8f07-4adbd0aa2226) |
There was a problem hiding this comment.
🟡 Changes recommended
A few small but concrete robustness/operability issues were found in the new sidecar/service plumbing (notably LLMLingua server boot assumptions and missing correlation header propagation to Bifrost).
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
open-sse/utils/upstreamResponseHeaders.ts:92
- getSidecarResponseHeaders uses a single allowlist for all three headers, which can incorrectly accept impossible combinations (e.g. X-Routing-Fallback-Reason: "bifrost") or allow X-Routed-By to carry fallback codes. Tightening the allowlist per header keeps the diagnostic surface consistent and avoids propagating misleading values.
src/shared/services/bifrost/bifrostExecutor.ts:47 - Bifrost dispatch currently constructs a dummy Request without propagating correlation headers from ExecuteInput.clientHeaders, so dispatchToBifrost can’t forward x-request-id / x-correlation-id for tracing. This also means sidecar logs/diagnostics won’t line up with the original request when debugging.
- Files reviewed: 40/40 changed files
- Comments generated: 1
- Review effort level: Lite
Remove obsolete changelog validation and untrack ignored planning artifacts so retained commit gates pass.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_4336e833-a8f9-4aa6-bdae-eace21fe0bf2) |
Merge Raudbjorn/OmniRoute dev at c4a59a4. Resolve docs-sync conflict by preserving removal of obsolete changelog validation; retain OpenAPI version checks.
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_94e5fa87-c4ea-4c70-bf67-438ec885aa08) |
There was a problem hiding this comment.
🔵 Needs a closer look
There are at least two correctness/safety issues to address (test env leakage across concurrent node:test runs, and overly broad allowlisting of sidecar diagnostic headers).
Review details
Suppressed comments (2)
Previously missed (2) — in code that hasn't changed since the last review.
open-sse/utils/upstreamResponseHeaders.ts:92
- getSidecarResponseHeaders uses the same allowlist for all three headers, which allows invalid combinations like
X-Routed-By: bifrost-errorto be propagated if an upstream sets it. Tighten the allowlist per header soX-Routed-Byonly ever forwardsbifrost, while fallback headers forward only valid fallback/reason codes.
tests/unit/services/installers/llmlingua.test.ts:10 - This test mutates global process.env (DATA_DIR/NODE_ENV/DISABLE_SQLITE_AUTO_BACKUP) at module scope and never restores it, which can leak into other node:test files (especially under --test-concurrency) and cause order-dependent failures. Snapshot the original env before these assignments and restore it in test.after (or use tests/_setup/isolateDataDir.ts to manage DATA_DIR isolation consistently).
- Files reviewed: 41/41 changed files
- Comments generated: 0 new
- Review effort level: Lite
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed correctness/robustness issues (LLMLingua sidecar request/schema mismatch, overly broad diagnostics header validation, and test env leakage under concurrent runs) that should be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
Suppressed comments (1)
tests/unit/services/installers/llmlingua.test.ts:66
- Follow-up to the module-scope env overrides: restore the original env values in the existing
test.after()cleanup so parallel test files don't observe a modifiedDATA_DIR/NODE_ENV.
- Files reviewed: 41/41 changed files
- Comments generated: 3
- Review effort level: Lite
Comment on lines
+83
to
+92
| export function getSidecarResponseHeaders(headers?: Headers): Record<string, string> { | ||
| const result: Record<string, string> = {}; | ||
| for (const name of ["X-Routed-By", "X-Routing-Fallback", "X-Routing-Fallback-Reason"]) { | ||
| const value = headers?.get(name); | ||
| if (value === "bifrost" || value === "bifrost-error" || value === "bifrost-cooldown") { | ||
| result[name] = value; | ||
| } | ||
| } | ||
| return result; | ||
| } |
Comment on lines
+52
to
+56
| schema = z.object({ | ||
| text: z.string().min(1).max(1_000_000), | ||
| model: z.string().max(200).optional(), | ||
| compressionRate: z.number().gt(0).lte(1).optional(), | ||
| }); |
Comment on lines
+7
to
+10
| const TEST_DATA_DIR = fs.mkdtempSync(path.join(os.tmpdir(), "omniroute-llmlingua-installer-")); | ||
| process.env.DATA_DIR = TEST_DATA_DIR; | ||
| process.env.NODE_ENV = "test"; | ||
| process.env.DISABLE_SQLITE_AUTO_BACKUP = "true"; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports upstream #12967 and #12953 onto this fork's
devbranch.LLMLingua now has supervised HTTP compression, lifecycle API routes, a disabled-by-default DB seed, and a Services dashboard tab. The upstream placeholder returned unchanged text while claiming compression; this port instead calls OmniRoute's existing ONNX worker, validates and bounds requests, and reports actual output. The existing optional ML runtime is required; installing the service wrapper does not download it.
Bifrost dispatch runs at the native executor step after authentication, policy, guardrails and admission. This corrects the upstream route-level authentication bypass while retaining normal response translation/accounting. OpenAI and Claude wire formats use their Bifrost protocol adapters. Network/5xx failures fall back to native dispatch with the shared cooldown, cancellation propagates, and credentials/sensitive response headers are not relayed. Explicit CLIProxyAPI/Dario overrides retain precedence.
Enablement:
BIFROST_INGRESS_ENABLED=1after configuring Bifrost's provider/model and existing Bifrost connection settings.LLMLINGUA_BASE_URL=http://127.0.0.1:20135to use HTTP compression. Unset keeps the existing direct worker path.Validation:
--no-stashoption to honor the repository's stash ban.devbase.check:docs-allstops at existing migration-count drift (base already has 170 migrations while docs claim 169; this port adds one). Broad API typecheck also remains red. No unrelated baseline fixes or agent-instruction edits are included.Keeping this draft while the broader repository checks remain red. Upstream base-red tracking: cheaper-inference#12732.
Thanks @rqzbeh for the upstream implementations; human authorship is retained in the commit.
Summary by Sourcery
Integrate optional LLMLingua compression and authenticated, fallback-capable Bifrost routing into the native request pipeline.
New Features:
Bug Fixes:
Enhancements:
Documentation:
Tests:
Chores:
Note
Medium Risk
Changes the native chat executor path and optional compression backend; Bifrost ingress is gated off by default but affects routing, fallbacks, and response headers when enabled.
Overview
Adds LLMLingua as a sixth supervised embedded service (default port 20135): dashboard tab, loopback lifecycle APIs, DB seed/migration, installer that writes a local HTTP server around the existing ONNX worker, and bootstrap registration without model sync. Prompt compression can optionally call that sidecar when
LLMLINGUA_BASE_URLis set; failures or bad responses fail open to the in-process worker.Introduces opt-in Bifrost ingress via
BIFROST_INGRESS_ENABLED=1: native-mode executors are wrapped so OpenAI/Claude wire formats dispatch through Bifrost after the normal chat pipeline, with shared routing/cooldown logic moved tosrc/shared/services/bifrost/, native fallback on errors/cooldown, and allowlistedX-Routed-By/X-Routing-Fallback*headers on streaming and non-streaming responses. Relay route modules re-export the shared Bifrost helpers instead of duplicating them.Docs/OpenAPI/env/release checklist and
check-docs-syncare updated (CHANGELOG sync removed for this fork). Broad unit/integration tests cover auth-before-Bifrost, header filtering, LLMLingua HTTP validation, and fallback behavior.Reviewed by Cursor Bugbot for commit 3010445. Bugbot is set up for automated code reviews on this repo. Configure here.