Skip to content

docs(use-cua-with): add aimlapi.com provider guide - #1

Merged
Lookoff-AIMLAPI merged 2 commits into
mainfrom
docs/aimlapi-provider-guide
Sep 9, 2026
Merged

Lookoff-AIMLAPI merged 2 commits into
mainfrom
docs/aimlapi-provider-guide

Conversation

@Lookoff-AIMLAPI

Copy link
Copy Markdown
Member

Summary

  • Problem this solves: the Cua Agent SDK already reaches aimlapi.com — LiteLLM ships an aiml provider, so ComputerAgent(model="aiml/<id>") resolves with no registration — but no page said so, and the two things that make a first attempt fail (loop selection by regex, and an unauthenticated model catalog) were undiscoverable.
  • What changed: one new page, docs/content/docs/use-cua-with/aimlapi.mdx, plus its entry in the section meta.json and a card in use-cua-with/index.mdx. No code changes.

Documentation only. The integration itself needs nothing: aiml/ is already a LiteLLM provider pointing at https://api.aimlapi.com/v1 and reading AIML_API_KEY.

Related work

Refs — none.

RFC: not required. rfcs/README.md: "An RFC is not required for small bug fixes, routine documentation changes, internal refactors that preserve public behavior, or urgent private security response." Nothing here touches a public SDK, CLI, MCP, protocol, or file-format contract.

Compatibility and risk

  • User-visible, API/CLI/MCP, migration, permission, or platform impact: one new docs route, /use-cua-with/aimlapi. No behavior change.
  • Risk and rollback: revert either commit independently.

Validation

Ran from docs/, on a pristine checkout first and again after the change:

Check Baseline After
pnpm docs:check-hygiene pass pass
pnpm docs:check-links 0 errored files, 0 errors 0 errored files, 0 errors
pnpm build exit 0, 132 static pages exit 0, 133 static pages
python -m pytest libs/python/agent/tests -q 49 passed 49 passed

Exit codes were read from the bare commands, not through a pipe.

Live inference, through ComputerAgent

Not a mock and not a raw curl. Each run constructed a real ComputerAgent with a dict-based computer handler (cua_agent.computers.CustomComputerHandler) whose screenshot() returned a real 1024x768 PNG containing a green button labelled CONTINUE at x 420-620, y 420-480. Prompt: "Look at the screen. Name the button label you see, then click it." Key supplied through AIML_API_KEY.

Model string Loop selected Result
aiml/z-ai/glm-4.5v Glm4vConfig I can see a Settings window with a green button labeled "CONTINUE". I'll click this button now. then {"type":"click","button":"left","x":526,"y":455}
aiml/alibaba/qwen3-vl-plus Qwen3VlConfig {"type":"click","button":"left","x":519,"y":452}
aiml/alibaba/qwen3-vl-flash Qwen3VlConfig {"type":"click","button":"left","x":518,"y":449}
aiml/openai/gpt-4o-mini GenericVlmConfig {"type":"click","button":"left","x":512,"y":499}
aiml/openai/gpt-5-5 GenericVlmConfig text CONTINUE

Three of the five landed inside the button rectangle and one read its label back verbatim, so the screenshot genuinely reached the model. Tool calling is exercised by the same runs: the computer action is returned as a function call against the loop's computer tool schema.

aiml/anthropic/claude-sonnet-4.5 was run too and is the reason for the caveat in the page. It selects AnthropicHostedToolsConfig, which put [{"type":"computer_20250124","function":{"name":"computer","parameters":{"display_height_px":768,"display_width_px":1024,"display_number":1}}}] on a /v1/chat/completions request. The endpoint answered 200 and ignored the tool, so no screenshot was ever sent (23 input tokens) and the model narrated an imaginary "Click Me" button with a pyautogui snippet. It fails silently rather than erroring. This is not aimlapi-specific: find_agent_config matches .*claude-.* against the whole model string, so openrouter/anthropic/claude-sonnet-4.5 selects the same loop. Left alone here — changing loop selection is a public-behavior change and belongs in its own issue.

Request shape on the wire

Every outbound body was captured at the httpx layer:

  • image parts are emitted as {"type":"image_url","image_url":{"url":"data:image/png;base64,..."}} — the detail key is absent, never null. That matters, because image_url.detail: null is rejected with 400 by this endpoint while an omitted detail and "auto" both return 200. Cua sets detail nowhere in the tree (grep for it under libs/python/agent/cua_agent/ returns nothing), so the vision path is safe as written.
  • top-level bodies carried only the keys actually in use (model, messages, and tools where a loop sends them); no key was serialised as null. The endpoint rejects null on tools, temperature, top_p, seed, response_format, stream and others, so this matters and currently holds.

Known gaps

  • Embeddings were not exercised because Cua does not use them. There is no litellm.embedding/aembedding call anywhere in libs/python/. Worth knowing anyway: LiteLLM 1.86.2 ships llms/aiml/chat/ and llms/aiml/image_generation/ but no embedding/ route, so an aiml/<model> embedding call would fail with "Unmapped LLM provider for this endpoint" if one were ever added.
  • The Codex row points at Codex's own configuration rather than a tested command; the repo has no tested Codex-plus-custom-endpoint recipe to link, and inventing one was out of scope.
  • pip install cua-agent[qwen] was not sufficient to run the default GenericVlmConfig loop in a clean venv: qwen-agent needs numpy, soundfile and python-dateutil at import time and qwen-vl-utils needs torch, none of which the extra pulls in. Unrelated to this change, so not touched, but it is the first wall a reader following the page will hit.

Contributor and release checks

  • The PR is focused and the description matches the final diff.
  • This change does not require an RFC, or the accepted RFC is linked above.
  • Tests, documentation, and platform evidence are included or the gap is explained.
  • The PR title is a Conventional Commit describing the production change.
  • External contributor authorship is preserved, or no external contribution is included.
  • If release-tracked files changed but this is intentionally non-releasing, the no-release label is applied — no release-tracked files changed.

Commit layout

Two commits, deliberately separated:

  1. docs(use-cua-with): add aimlapi.com provider guide — the page and its registration, in the position a new provider would naturally take (appended after minimax).
  2. chore(aimlapi): fork-only placement — do not send upstream — moves the entry to the front of the sidebar group and the card grid. Drop this commit before offering the guide upstream. No badge was invented; this docs theme has no featured or recommended concept, so only the order changes.

The Agent SDK already reaches aimlapi.com: LiteLLM ships an `aiml`
provider, so `ComputerAgent(model="aiml/<id>")` resolves with no
registration, but nothing said so and readers had no way to discover the
prefix, the `AIML_API_KEY` name, or which ids are safe to pass.

The loop-selection caveat is the part that costs people a debugging
session. Loops are chosen by regex over the entire model string, so
`aiml/anthropic/claude-*` matches `.*claude-.*` and lands on the
Anthropic hosted-tools loop. That loop sends a `computer_20250124` tool
to a chat-completions endpoint, which returns 200 and ignores it, so the
run neither errors nor sends a screenshot -- it just narrates. Naming ids
that reach a chat-completions loop keeps a first attempt from failing
silently.

The catalog note exists for the same reason: `GET /v1/models` answers
without authentication, so it cannot be used to check a key.
Both lists in this section are hand-ordered by when each provider landed,
not alphabetically, so position is editorial rather than mechanical. This
moves the aimlapi.com entry to the front of the sidebar group and the
"Models and providers" card grid.

There is no featured or recommended badge in this docs theme, so nothing
is invented; only the order changes. Kept as a separate trailing commit
so it can be dropped before the guide is offered upstream.
@Lookoff-AIMLAPI
Lookoff-AIMLAPI merged commit 37febc3 into main Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants