Conversation
…n endpoint Capture descriptions could only use OpenRouter, Vertex or the Gemini API, none of which is reliably reachable from mainland China or able to target a self-hosted model. Setting HYPERFRAMES_VISION_BASE_URL, HYPERFRAMES_VISION_API_KEY and HYPERFRAMES_VISION_MODEL together now routes captioning through that endpoint ahead of the other providers. Setting only some of them warns with the missing names, skips captioning and reports the phase as degraded instead of falling back to another provider. OpenRouter goes through the same request helper with its URL, auth header and error text unchanged. Signed-off-by: FenjuFu <fufenjupku@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
hyperframes capturecan describe images through any OpenAI-compatible vision endpoint. SetHYPERFRAMES_VISION_BASE_URL,HYPERFRAMES_VISION_API_KEYandHYPERFRAMES_VISION_MODELtogether and that endpoint is used ahead of OpenRouter, Vertex and Gemini.Why
OpenRouter, Vertex and the Gemini API are all hard to reach from mainland China, and none of them can target a self-hosted model. Those users get catalog-fallback descriptions on every capture even when they have a working vision endpoint.
Related work
Closes #4846
Picks up #1809 by @yuemeng200, closed for merge conflicts after a review that endorsed the approach. This PR is rebuilt on current
main, which since gained Vertex (#3561) in the same function, and addresses that review:degraded / internal-error, the same as an unparseable Vertex service account. It never falls back to OpenRouter or Gemini.providerNameas an argument:openAiCompatibleCaptionOne(providerName, endpoint)is self-contained. OpenRouter now goes through it too, and its URL (https://openrouter.ai/api/v1/chat/completions),Bearerheader, request body and error text (OpenRouter request failed with HTTP <status>) are unchanged. The existing OpenRouter tests pass without edits.How
resolveCustomVisionEndpoint()returnsunset,incompletewith the missing names, orreadywith the endpoint.captionImagesWithGeminiandpostExtractionPhase.hasVisionCredentialsboth use it, so theasset-descriptions.mdheader can't claim vision credentials for a half-set endpoint./chat/completions.asset-descriptions.mdand the docs (guides/authentication.mdx,packages/cli.mdx) mention the new variables. The docs say to pick a model that accepts image input and to give servers that ignore auth any non-empty key.Test plan
max_tokens, image data URI); priority over OpenRouter and Gemini; failed request counted as a provider failure; key and model without a base URL withGEMINI_API_KEYset (no request, no Gemini client, warning names the variable and does not contain the key, phase degraded); base URL alone withOPENROUTER_API_KEYset;resolveCustomVisionEndpointunset and incomplete cases.captionImagesWithGeminifrom this branch with the three variables pointed at iFlytek Astron's Token Plan, a live OpenAI-compatible endpoint (https://maas-token-api.cn-huabei-1.xf-yun.com/v2/, trailing slash included), on two PNGs fromdocs/images/image-thumbnail-evidence/:xopkimik26(Kimi-K2.6): 2/2 captioned, e.g. "A dark gray webpage displays white text reading "Timeline JPEG thumbnail resource check" above a horizontal strip of synthetic rainbow-colored JPEG test bars."; outcome all zeroxopqwen35397b(Qwen3.5-397B): 2/2 captioned; outcome all zeroGEMINI_API_KEYset: no request, warning… must be set together; missing HYPERFRAMES_VISION_BASE_URL., outcomeinternalError: trueLocal checks:
vitest run src/capture/contentExtractor.test.ts src/capture/contentExtractor.file-race.test.tspasses 36/36. The 7 new tests fail againstmain'scontentExtractor.ts.oxlintandoxfmt --checkare clean on the changed files, and so arecheck-comment-citations.mjsandcomment-ratchet.mjs.tscover the changed files reports nothing in them. I ran this on Windows without building@hyperframes/core, so the only errors were its generatedruntime-inlinemodules.