Repository navigation
Default onboarding LLM to OpenAI GPT-5.5 - #1103
Conversation
Co-authored-by: openhands <openhands@all-hands.dev>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
🔍 Review in progress… We are performing the review through OpenHands Cloud Automation. You can log in and view the conversation here. |
|
✅ Review complete. This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
Code Review: Default Onboarding LLM to OpenAI GPT-5.5
Overview
This PR changes the default LLM model in the onboarding flow from Anthropic Claude Opus (anthropic/claude-opus-4-8) to OpenAI GPT-5.5 (openai/gpt-5.5). The changes span configuration, UI text, tests, and mock data — all consistent and cohesive.
Comments
🟡 Suggestion — translation.json: The subtitle changes are syntactically correct, but consider whether the tone fits the onboarding experience. "We've pre-selected OpenAI GPT-5.5" may feel abrupt to new users. Something like "We've selected OpenAI GPT-5.5 as a recommended default" or "Your recommended starting point: OpenAI GPT-5.5" could feel more inviting. However, this is a stylistic choice — current wording is acceptable.
🟠 Important — test robustness: The test pre-fills the LLM step with OpenAI GPT-5.5 hardcodes the expected model string. If the model ID ever changes (e.g., openai/gpt-5.5-20250601), the test will fail. Consider using a constant or extracting this from the same ONBOARDING_LLM_OVERRIDES object under test, or at minimum add a comment noting this dependency.
Summary
✅ LGTM with minor suggestions. The implementation is clean, consistent, and well-tested. The core logic change is straightforward and the changes to supporting files (tests, mocks, translations) are appropriate. No blockers — the suggestions above are improvements to consider but don't block approval.
This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation
Co-authored-by: openhands <openhands@all-hands.dev>
|
Addressed the OpenHands bot feedback in 548168c by sharing the onboarding default LLM model constant between the implementation and test, and resolved both review threads. Ready for another review. This comment was generated by an AI agent (OpenHands) on behalf of the user. |
|
✅ Review complete. This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
Code Review: Default Onboarding LLM to OpenAI GPT-5.5
Overview
This PR changes the default LLM model in the onboarding flow from Anthropic Claude Opus to OpenAI GPT-5.5. The changes span configuration, UI text, tests, and mock data.
Issue Found
🟠 config/defaults.json (line 7) — This PR downgrades the automation version from 1.0.0a6 (on main) to 1.0.0a5. This appears to be unintended drift from the merge with main. If the intent is to stay on 1.0.0a5, this should be justified; otherwise, it should be updated to 1.0.0a6 to match main.
Also, the new test in __tests__/scripts/dev-with-automation.test.ts that asserts DEFAULT_AUTOMATION_VERSION === "1.0.0a5" will need to be updated if the version is corrected to match main.
Summary
✅ LGTM with one issue to address. The core change (GPT-5.5 default) is well-implemented with proper test coverage and translated UI text. The version downgrade in config/defaults.json needs clarification or correction.
This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation
…-gpt55-default Co-authored-by: openhands <openhands@all-hands.dev>
|
Merged the latest This comment was generated by an AI agent (OpenHands) on behalf of the user. |
|
✅ Review complete. This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here. |
all-hands-bot
left a comment
all-hands-bot
left a comment
There was a problem hiding this comment.
Code Review: Default Onboarding LLM to OpenAI GPT-5.5
Overview
This PR changes the default LLM model in the onboarding flow from Anthropic Claude Opus to OpenAI GPT-5.5. The changes span configuration, tests, mock data, and internationalization.
Changes Summary
| File | Change |
|---|---|
src/components/features/onboarding/steps/setup-llm-step.tsx |
Exports a shared constant ONBOARDING_DEFAULT_LLM_MODEL and uses it in the overrides |
__tests__/api/option-service.test.ts |
Updates model and provider assertions to include GPT-5.5 |
__tests__/components/onboarding/onboarding-modal.test.tsx |
Imports the shared constant and references it in the test |
src/mocks/settings-handlers.ts |
Adds GPT-5.5 to mock model lists |
src/i18n/translation.json |
Updates LLM subtitle across 13 languages |
Assessment
- Clean extraction of the model constant to avoid duplication between implementation and tests
- All 13 language translations updated consistently
- Mock handlers updated to include the new model
- Both test files updated appropriately
This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation.
❌ Mock-LLM Docker E2E Test Results19/25 passed · 1 failed · 5 skipped Commit:
🔍 Failure details (1)❌ mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
Earlier mock-LLM tests (conversation, automation) configure the real agent-server with llm_base_url pointing at the mock LLM server. That stale URL causes LlmSettingsScreen to open in Advanced view (text inputs) instead of Basic view (ModelSelector dropdowns), so the onboarding regression test can't find the provider/model inputs. Intercept GET /api/settings in routeOnboardingLlmCatalog to clear llm_base_url, ensuring the form renders in Basic view. Co-authored-by: openhands <openhands@all-hands.dev>
Document the workflow for diagnosing CI failures: reading PR comments, downloading artifacts, inspecting error-context.md page snapshots, and common failure patterns (stale state, view mode mismatch, route interception). Co-authored-by: openhands <openhands@all-hands.dev>
|
| Status | Test | Duration |
|---|
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)
|
| Status | Test | Duration |
|---|
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses)
❌ Mock-LLM E2E Tests24/25 passed · 1 failed Commit:
🔍 Failure details (1)❌ mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
|
@OpenHands why did this test fail? investigate and fix: #1103 (comment) Then post a comment on github with your analysis and fix, mention me in it so I can see it. |
|
I'm on it! enyst can track my progress at all-hands.dev |
Move the settings interception from the shared helper into the specific test's beforeGoto callback. Clear agent_settings.llm.base_url (not top-level llm_base_url) from the real server response so the LLM form renders in Basic view with ModelSelector dropdowns. The interception is registered AFTER routeSessionApiKey so Playwright's LIFO order processes it first for /api/settings requests. Co-authored-by: openhands <openhands@all-hands.dev>
❌ Mock-LLM Docker E2E Test Results19/25 passed · 1 failed · 5 skipped Commit:
🔍 Failure details (1)❌ mock-llm-onboarding-regressions.spec.ts › onboarding recent regressions › defaults the LLM setup step to OpenAI GPT-5.5Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
✅ Mock-LLM E2E Tests25/25 passed Commit:
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
🔶 Mock-LLM Docker E2E Test Results20/25 passed · 5 skipped Commit:
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
|
@/tmp/pr-1103-comment.md |
|
@/tmp/pr-1103-correction.md |
|
Since my last summary, there were no new code changes or GitHub actions. Final status:
|
|
🚀 Released in v1.2.0. |
✅ Mock-LLM E2E Tests25/25 passed Commit:
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
🔶 Mock-LLM Docker E2E Test Results20/25 passed · 5 skipped Commit:
Posted by the Mock-LLM E2E workflow · results are deterministic (scripted LLM responses) |
📸 Snapshot Test ReportWarning Snapshot comparison step crashed (timeout, OOM, or runner error) — diff results below may be incomplete or absent. ✅ No baseline found on main — all 73 snapshots are new and will become the baseline once this PR merges.
🆕 New snapshots (73)These snapshots have no baseline on main and will become the new baseline once this PR merges.
|
This branch was successfully deployed
4 participants
Summary
openai/gpt-5.5Verification
npx vitest run __tests__/api/option-service.test.ts __tests__/components/onboarding/onboarding-modal.test.tsx __tests__/api/mock-settings-handlers.test.ts __tests__/api/config-service.test.tsnpm run typechecknpm run buildnpm testThis PR was created by an AI agent (OpenHands) on behalf of the user.
🐳 Docker images for this PR
• GHCR package: https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas
ghcr.io/openhands/agent-canvasghcr.io/openhands/agent-server:1.24.0-pythonopenhands-automation==1.0.0a681ba0f276822311bfcc7f3689c610acd0359e7b2Pull (multi-arch manifest)
# Multi-arch manifest — Docker automatically pulls the correct architecture docker pull ghcr.io/openhands/agent-canvas:sha-81ba0f2Run
All tags pushed for this build
About Multi-Architecture Support
sha-81ba0f2) is a multi-arch manifest supporting both amd64 and arm64sha-81ba0f2-amd64) are also available if needed