Suggested Documentation Changes from August 2026 Analytics Audit

[DevinVinson]

Suggested Documentation Changes for OpenHands/docs

Prepared: September 8, 2026
Evidence window: July 28–August 27, 2026 (31 days)
Inputs: August Mintlify chat, search, page-view, and feedback exports; August chat, conversation, and GitHub cross-source analyses; current OpenHands/docs reviewed September 8.

Decision summary

July’s P0 work addressed broad onboarding gaps: Agent Canvas now has a connected lifecycle and recovery route, a clearer operating model, a task-first Skills overview, and an LLM Profiles configuration/recovery workflow. August validates maintaining those routes but changes the investment priority:

1. Make provider configuration and recovery a direct, task-first journey.
2. Audit and improve OpenRouter first, using concentrated negative feedback as the entry point.
3. Improve broad search routing for model, local, and agent intent.
4. Link Agent Canvas onboarding to skills, project guidance, subagents, and automations.
5. Treat Docker and first-run recovery as release-validated maintenance work.

The August comparison window contains 3,949 valid queries across 1,719 conversations, down 10.9% and 13.1% from July. Answered-query rate improved from 94.4% to 97.1%. Provider configuration was the only major technical cohort to grow in absolute volume, while skills and agents remained the largest broad-interest area.

Chat cohorts overlap and are directional. Views and feedback use the July 28–September 4 export because supplied page-view data has no row-level date. Mintlify category coverage worsened in August, so recommendations use query-text cohorts and direct feedback alongside categories.

Recommended work, in order

Priority	Change	Evidence	Primary docs targets	Success signal
P0	Build a provider configuration and recovery path	560 provider rows, up 6.5% while most large cohorts declined; 12 unanswered provider rows; 16 August GitHub issues labeled llm	New task-first provider decision guide; LLM Profiles; troubleshooting; LLM overview	Higher CTR for llm, model, provider, local, and api; fewer provider follow-ups and unanswered queries
P0	Audit and rewrite OpenRouter guidance	4 unhelpful votes on /openhands/usage/llms/openrouter; current guide is a short field list	openhands/usage/llms/openrouter.mdx	Improved helpful/unhelpful ratio; fewer configuration and model-ID failures
P1	Tune broad provider and local search landing paths	local: 39 searches / 48.7% CTR; api: 42 / 66.7%; open: 38 / 2.6%; ollama: 47 / 89.4%	LLM overview, Local LLMs, LLM Profiles, frontmatter, and internal links	Better CTR and relevant top results for broad terms
P1	Create task-oriented agent/customization entry points	Skills/agents: 631 rows across 450 conversations (16.0%); agent: 34 searches / 11.8% CTR	Canvas overview, Skills overview, first-time setup, and Customize/Skills links	Higher CTR for agent, skill, subagent, and AGENTS.md
P1	Keep setup, Docker, and backend recovery release-validated	361 setup rows; 313 Docker/self-hosting rows; Docker had 2 unanswered rows and 61.9% CTR	Canvas setup, troubleshooting, Docker, and local-backend guides	Stable or improving recovery answer rate and feedback after releases
P2	Add a lightweight release-to-docs impact review	64 Agent Canvas issues, 69 frontend issues, 126 fix PRs, and 7 docs PRs	PR/release workflow or template, after ownership is agreed	User-facing changes have a docs link or explicit no-impact rationale

P0 — Build a provider configuration and recovery path

Create an Agent Canvas decision guide organized by what access users have, rather than provider name alone:

I have	Configure through	Include
Direct provider API key	Basic LLM profile	Provider/model selection, key, save, and verification
OpenHands LLM API key	Basic LLM profile	OpenHands provider, model selection, scope, and verification
Local OpenAI-compatible server	Advanced LLM profile	Exact served model ID, base URL reachable from the backend, host/container networking, and verification
LiteLLM proxy	Advanced LLM profile	litellm_proxy/ prefix, proxy URL/key, and proxy model-name match
ACP subscription login	ACP agent setup	Same-machine login prerequisite and credential recovery

Link from this route to existing Claude/Anthropic, Ollama, LiteLLM, OpenAI/Codex, LM Studio, Qwen, and OpenRouter references. It complements—not replaces—provider-specific pages.

Add a shared symptom-to-fix map for unrecognized providers, invalid model IDs, unreachable local endpoints, authentication/permission errors, and model context/tool-use mismatch. In each case, give the first check and a decisive next action.

P0 — Audit and improve OpenRouter first

Expand openhands/usage/llms/openrouter.mdx into a concise setup and recovery page:

• Explain when OpenRouter is appropriate versus direct provider access, a local endpoint, or LiteLLM proxy.
• State the Basic versus Advanced Agent Canvas profile path.
• Explain openrouter/<model-id> and how to obtain a current model ID.
• Include API-key prerequisites and a verification step.
• Cover invalid model ID, unavailable model, authentication, and endpoint/model mismatch.
• Link LLM Profiles, the decision guide, model troubleshooting, and the OpenRouter model catalog.

Validate documented configurations against the current supported product surface before publication; do not add unverified capability claims.

P1 — Search and customization routing

For provider/local search, put task choices on the first screen of the LLM overview: API key, local OpenAI-compatible endpoint, LiteLLM proxy, or local runtime. Add a Local LLMs disambiguation block: local model, local Agent Canvas backend, or local workspace/files. Improve task-language titles, descriptions, headings, and links for LLM, model, provider, local, Ollama, and API, then validate production search after publication.

For agent/customization intent, retain the July Skills overview as canonical. From Canvas overview and first-time setup, route users to AGENTS.md, reusable skills, subagents/delegation, plugins, MCP, or automations. Ensure broad agent intent lands on a general OpenHands/Agent Canvas route before ACP-specific guidance.

P1 — Maintain setup, Docker, and recovery content

Docker remains high-volume but comparatively well-covered. Do not expand generic Docker documentation before provider work. For each relevant release, validate install/restart/update/uninstall/reset commands; browser, backend, key, port, and model recovery links; Docker trust-boundary guidance; and version-specific behavior that needs a migration cue or release note.

P2 — Establish release-to-docs impact review

Propose a lightweight check for Canvas, LLM, frontend/UI, Docker/backends, MCP/integrations, and automation changes:

1. Link affected docs and examples, or explain why no user-facing documentation change is needed.
2. For fixes/behavior changes, decide whether troubleshooting, migration, or release-note content needs updating.
3. Confirm documented configurations against the shipped release.

Choose ownership and workflow location before implementation; this issue records the need, not a prescribed process change.

Delivery sequence

1. Validate and update OpenRouter guidance.
2. Add the provider decision/recovery route and error map; connect LLM Profiles, Local LLMs, LiteLLM, and troubleshooting.
3. Tune broad provider/local search landing paths and validate production results.
4. Add Canvas-to-skills, subagent, project-instruction, plugin, MCP, and automation routing without duplicating canonical pages.
5. Define a docs-impact review and revalidate high-traffic operational guides during releases.

Measurement plan

Review the next full analytics window directionally:

• Chat: provider unanswered rate, two-turn provider troubleshooting sessions, and skills/agent discovery cohorts.
• Search: hits, CTR, and top-result relevance for llm, model, provider, local, api, agent, skill, and subagent.
• Feedback: helpful/unhelpful votes for OpenRouter, LLM Profiles, Canvas setup, and overview.
• Release quality: user-facing Canvas/LLM/integration changes with a documented impact decision; validate copyable configurations before publication.

This issue was created by an AI agent (OpenHands) on behalf of the user.