test: add onboarding regression E2E coverage

[enyst]

Why

Recent issue/PR triage over the last 48 hours turned up several regressions that should stay covered by E2E tests. The static/public auth regressions already have mock-LLM E2E coverage, so this PR adds mock-LLM browser coverage for onboarding regressions that were only covered at component level:

• OpenHands/OpenHands#15696 / fix(frontend): keep modal open on errant outside click #1100: backdrop click or Escape must not dismiss onboarding and permanently mark it complete.
• [Bug]: Default model provider is OpenHands instead of Anthropic #2097 / fix(frontend): default LLM setup to Anthropic Claude Opus 4.8 #1089: first-run LLM setup should default to Anthropic Claude Opus instead of the OpenHands provider.

Summary

• Added tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts with two Playwright regression tests for the onboarding dismissal and LLM default flows.
• Updated the onboarding snapshot spec to use the current backend-first onboarding order and the home route where OnboardingHost mounts.
• Moved the new regression coverage into the mock-LLM suite after review feedback, with deterministic model/provider catalog routing for the onboarding UI assertion.

Test Plan

• npm run typecheck
• npx eslint tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts tests/e2e/snapshots/onboarding.snapshot.spec.ts
• MOCK_LLM_SESSION_API_KEY=mock-llm-onboarding-key MOCK_LLM_PYTHON=.mock-llm-venv/bin/python3 npx playwright test --config=playwright.mock-llm.config.ts tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts
• npx playwright test tests/e2e/snapshots/onboarding.snapshot.spec.ts --project=chromium --update-snapshots --reporter=list

---

This PR was created by an AI agent (OpenHands) on behalf of the user.

---

🐳 Docker images for this PR

• GHCR package: https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas

Component	Value
Image	ghcr.io/openhands/agent-canvas
Architectures	amd64, arm64
Agent Server	ghcr.io/openhands/agent-server:1.24.0-python
Automation	openhands-automation==1.0.0a6
Commit	de379529d6444ebae0588c7997ff575920c3ea47

Pull (multi-arch manifest)

# Multi-arch manifest — Docker automatically pulls the correct architecture
docker pull ghcr.io/openhands/agent-canvas:sha-de37952

Run

docker run -it --rm \
  -p 8000:8000 \
  ghcr.io/openhands/agent-canvas:sha-de37952

All tags pushed for this build

ghcr.io/openhands/agent-canvas:sha-de37952-amd64
ghcr.io/openhands/agent-canvas:e2e-recent-regressions-amd64
ghcr.io/openhands/agent-canvas:pr-1105-amd64
ghcr.io/openhands/agent-canvas:sha-de37952-arm64
ghcr.io/openhands/agent-canvas:e2e-recent-regressions-arm64
ghcr.io/openhands/agent-canvas:pr-1105-arm64
ghcr.io/openhands/agent-canvas:sha-de37952
ghcr.io/openhands/agent-canvas:e2e-recent-regressions
ghcr.io/openhands/agent-canvas:pr-1105

About Multi-Architecture Support

• Each tag (e.g., sha-de37952) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., sha-de37952-amd64) are also available if needed

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
agent-canvas	Ready	Preview, Comment	Jun 3, 2026 10:21pm

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

Code Review: test: add onboarding regression E2E coverage

Good addition of regression coverage for the recent onboarding fixes. The tests are well-structured and cover the specific bugs mentioned (#1085/#1100 and #1077/#1089).

Test structure: The regressions/ directory is the right home — these are bug-coverage tests, not snapshot or mock-LLM integration tests. Existing review comment on directory location noted but addressed below.

Minor suggestions:

🟡 Suggestion: The advanceToLlmStep function advances through steps 0→1→2 but the test for LLM defaults only validates the final step. The function could also benefit from a comment explaining which step indexes correspond to which UI flows, since step numbering was recently changed.

[all-hands-bot]

Code Review: test: add onboarding regression E2E coverage

Good addition of regression coverage for the recent onboarding fixes. The tests are well-structured and cover the specific bugs mentioned (#1085/#1100 and #1077/#1089).

Test structure: The regressions/ directory is the right home — these are bug-coverage tests, not snapshot or mock-LLM integration tests. Existing review comment on directory location noted but addressed below.

Minor suggestions:

🟡 Suggestion: The advanceToLlmStep function advances through steps 0→1→2 but the test for LLM defaults only validates the final step. The function could also benefit from a comment explaining which step indexes correspond to which UI flows, since step numbering was recently changed.

[enyst]

Addressed review feedback in 0a6566b: moved the onboarding regression spec into the mock-LLM suite, clarified onboarding step-order comments, and resolved the review threads. Ready for another look.

This comment was generated by an AI agent (OpenHands) on behalf of the user.

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation.

[all-hands-bot]

Code Review: test: add onboarding regression E2E coverage

Good addition of regression coverage. The tests are well-structured and directly address the issues mentioned in the PR description.

Summary of Changes

New file: tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts

• Regression test for #1085/#1100: Modal stays open on backdrop click and Escape key press. Also verifies openhands-onboarded is not prematurely set.
• Regression test for #1077/#1089: LLM setup step defaults to Anthropic Claude Opus 4.8, not OpenHands provider.

Modified file: tests/e2e/snapshots/onboarding.snapshot.spec.ts

• Updated to reflect the new backend-first onboarding order (step 0 = backend check, step 1 = agent selection)
• Changed route from /conversations to / where OnboardingHost mounts
• Renamed snapshot files to match new step order

Observations

1. Step order documentation - The explicit step-order comments (lines 85-87) are helpful and make the test fail clearly if the onboarding flow changes again.

2. Clean localStorage handling - The test properly removes openhands-onboarded to ensure onboarding modal appears, and the skip button correctly sets it to 1.

3. Mock LLM catalog routing - The deterministic model/provider catalog routing ensures stable assertions.

4. Consistent with existing patterns - The test follows the same structure as other mock-LLM specs (serial mode, helper imports from ./utils/mock-llm-helpers).

5. Minor: local waitForStep helper - Previous review feedback noted this could be extracted. Keeping it local is fine given it is only used here, and the step-order comments mitigate the duplication concern.

Verdict

LGTM - The regression coverage is appropriate and well-implemented.

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

Code Review: test: add onboarding regression E2E coverage

Overview

Good regression test coverage for recent onboarding fixes. The tests directly address the issues mentioned in the PR description (#1085 and #1077). Here are my observations:

Strengths

1. Clear test structure: The regression tests follow a consistent pattern with showOnboarding() helper and waitForStep() assertions.

2. Explicit step indexing: The comment in advanceToLlmStep() explaining the onboarding order is helpful for future maintainability. Good documentation of the recent reordering.

3. Robust assertions: Using .poll() for localStorage checks ensures the tests wait for actual state changes rather than relying on timing.

4. Snapshot test updates: The snapshot tests correctly reflect the new step ordering (backend check now at step 0).

Minor Suggestions

1. Code duplication in helpers: showOnboarding(), routeOnboardingLlmCatalog(), and advanceToLlmStep() are defined in both test files. Consider extracting these into a shared helper module (e.g., tests/e2e/mock-llm/onboarding-helpers.ts) to avoid duplication and ensure consistency.

2. Test isolation: The advanceToLlmStep() function assumes the backend health probe resolves quickly. While the test does wait for onboarding-backend-connected, the 10s timeout should be sufficient, but adding a comment explaining this assumption would help future readers.

3. Error messages: Consider adding more descriptive failure messages to the assertions, especially for the localStorage checks, to make debugging easier if tests fail in CI.

Testing Note

The label update-snapshots suggests the snapshots may need to be regenerated. Ensure the snapshots are updated before merging this PR. The visual snapshots should be validated to ensure they capture the correct UI state for each step.

Conclusion

These tests provide solid regression coverage for the onboarding fixes. The implementation is clean and well-documented. The suggestions above are minor improvements rather than blockers.

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

[enyst]

@OpenHands fix these please #1105 (review) and /iterate

[openhands-ai]

Since my last summary, there are no additional code changes or actions to report.

Checklist:

• ✅ The PR review feedback was addressed.
• ✅ Changes were kept focused and concise; no extraneous tracked changes remain.
• ✅ The PR branch was pushed and brought up to date with main.
• ✅ Local targeted verification passed.
• ✅ GitHub checks completed successfully with no pending or failing checks.
• ✅ Review threads are resolved / no unresolved threads remain.
• ⚠️ The only remaining blocker is external: GitHub still reports REVIEW_REQUIRED, so a maintainer approval is needed before merge.

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

Code Review: test: add onboarding regression E2E coverage

Summary

This PR adds end-to-end regression test coverage for two recently-fixed onboarding bugs:

1. #1085/#1100: Backdrop clicks and Escape key presses should not dismiss the onboarding modal or mark it as complete
2. #1077/#1089: First-run LLM setup should default to Anthropic Claude Opus instead of OpenHands

The changes are focused and well-scoped, adding 252 lines across 3 files with no unrelated modifications.

---

Files Reviewed

tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts (+120 lines)

Strengths:

• Good use of test.describe.configure({ mode: "serial" }) to prevent test interference
• The showOnboarding() helper is well-structured and properly seeds all required localStorage keys
• Test 1 properly verifies both the modal visibility and the openhands-onboarded state remains null after errant interactions
• Test 2 correctly checks both the provider AND model defaults

Suggestions:

• The test for "keeps the modal open on backdrop click and Escape" could benefit from also testing backdrop click specifically (not just coordinate click at 8,8), but the current approach is reasonable for a regression test
• The model value check claude-opus-4-8 is correct for the regression, but consider adding a comment noting this is the Anthropic model ID format

tests/e2e/snapshots/onboarding.snapshot.spec.ts (+52/-68 lines)

Strengths:

• Good use of networkidle wait state before snapshot capture
• The step-by-step navigation pattern is clear and well-documented in comments
• Using dispatchEvent("click") is appropriate for the clipped slide navigation issue

Suggestions:

• The LLM step wait could be more resilient by waiting for a specific element instead of networkidle which can be timing-dependent

tests/e2e/support/onboarding-helpers.ts (+80 lines)

Strengths:

• Clean, well-documented helper functions
• routeOnboardingLlmCatalog() properly mocks all three endpoints needed for the LLM step
• Good timeout values (15s for step waits, 10s for element visibility)

Suggestions:

• The advanceOnboardingToLlmStep() function is a nice addition for test clarity

---

Test Coverage Assessment

Regression	Test Coverage	Status
#1085/#1100 - Backdrop/Escape dismissal	✅ Covered	Test verifies modal stays open and openhands-onboarded is not set
#1077/#1089 - Anthropic default	✅ Covered	Test verifies both provider (Anthropic) and model (claude-opus-4-8) defaults

---

Quality Notes

• The PR description clearly explains the motivation and links the regressions
• All mocks are properly isolated to the mock-llm test environment
• The snapshot tests update is a good sign this PR has been properly vetted

---

Conclusion

This is a solid addition of regression test coverage. The tests are well-structured and properly isolate the tested behavior. No blocking issues identified.

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

[enyst]

Addressed the latest AI review suggestions in 95780be: added a note clarifying the Anthropic model ID assertion and replaced the onboarding snapshot tests' networkidle waits with a specific LLM settings readiness helper.\n\nLocal checks run again: targeted ESLint, npm run typecheck, mock-LLM onboarding regression spec, and onboarding snapshot spec.\n\n_This comment was generated by an AI agent (OpenHands) on behalf of the user._

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

Code Review: Onboarding Regression E2E Coverage

Good addition of regression tests covering the recent onboarding fixes. The test helper structure is clean and reusable.

Summary

File	Changes	Notes
tests/e2e/support/onboarding-helpers.ts	+91 lines	Shared helper functions for onboarding E2E tests
tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts	+122 lines	Regression tests for #1085, #1077
tests/e2e/snapshots/onboarding.snapshot.spec.ts	+55/-72	Updated step order and routes

🟡 Suggestion: Consider extracting showOnboarding to helpers

The showOnboarding function in mock-llm-onboarding-regressions.spec.ts is 38 lines of localStorage setup + route configuration. This pattern will likely be needed by other test files. Moving it to onboarding-helpers.ts would improve reuse.

// In tests/e2e/support/onboarding-helpers.ts

export async function showOnboarding(page: Page, apiKey: string) {
  await page.addInitScript(
    ({ apiKey }) => {
      // ... localStorage setup
    },
    { apiKey },
  );
  await routeSessionApiKey(page);
  await routeOnboardingLlmCatalog(page);
  await page.goto('/', { waitUntil: 'domcontentloaded' });
  await expect(page.getByTestId('onboarding-modal')).toBeVisible({
    timeout: 20_000,
  });
}

🟡 Suggestion: Add regression test for #1085 - Continue from previous state

The 'keeps the modal open on backdrop click and Escape' test covers dismissing, but consider adding a complementary test that verifies clicking outside before any step is interacted with also doesn't mark onboarding complete (the original bug manifested on very first load).

✅ Design Notes (acceptable)

• The step constants (ONBOARDING_BACKEND_STEP, etc.) are 0-indexed integers matching data-current-step, which is consistent
• Using dispatchEvent('click') instead of .click() is the right call for absolutely-positioned slides
• The networkidle wait has been replaced with waitForOnboardingLlmSettingsReady - this is more deterministic

Minor: Route change comment

The snapshot tests now go to '/' instead of '/conversations'. This is intentional for the new onboarding flow, but worth confirming this doesn't affect other test suites that may assume the /conversations entry point.

---

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

[enyst]

Addressed the follow-up AI review in de37952: moved the onboarding modal setup into the shared onboarding helper, and clarified that the dismissal regression test exercises the first-load path before any step interaction.\n\nLocal checks run again: targeted ESLint, npm run typecheck, mock-LLM onboarding regression spec, and onboarding snapshot spec.\n\n_This comment was generated by an AI agent (OpenHands) on behalf of the user._

[all-hands-bot]

✅ Review complete.

This review was performed through OpenHands Cloud Automation. You can log in and view the conversation here.

[all-hands-bot]

PR Review: test: add onboarding regression E2E coverage

Overview

This PR adds mock-LLM browser-based E2E coverage for two onboarding regressions:

• #1085/#1100: Backdrop click or Escape must not dismiss onboarding or mark it complete
• #1077/#1089: First-run LLM setup should default to Anthropic Claude Opus (not OpenHands provider)

Files changed:

• tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts (new, 95 lines)
• tests/e2e/snapshots/onboarding.snapshot.spec.ts (modified, +55/-72 lines)
• tests/e2e/support/onboarding-helpers.ts (new, 135 lines)

---

Code Review

Strengths

1. Clear regression documentation: Both tests include inline comments referencing the GitHub issues/PRs they cover, making the intent self-documenting.

2. Meaningful assertions: Tests use descriptive messages in expect() calls (e.g., "onboarding modal should ignore backdrop clicks and Escape"), which helps debugging when tests fail.

3. Proper test isolation: Using localStorage.removeItem("openhands-onboarded") in showOnboarding() ensures each test starts fresh.

4. Deterministic mocking: The routeOnboardingLlmCatalog() helper provides predictable LLM provider/model responses, avoiding flakiness.

5. Appropriate test mode: Using test.describe.configure({ mode: "serial" }) is the right choice for onboarding flow tests that share localStorage state.

6. Good helper abstractions: Moving common helpers to onboarding-helpers.ts reduces duplication and centralizes onboarding-specific logic.

7. Step reordering is well-integrated: The snapshot spec update correctly reflects the backend-first onboarding order.

---

Minor Suggestions

1. Potential race in advanceOnboardingToLlmStep(): The function advances through steps without waiting for the backend health probe to settle. The caller in the regression test does wait separately via advanceOnboardingToLlmStep(page) followed by waitForOnboardingBackendConnected(page) before advanceOnboardingToLlmStep(page). However, the internal waitForOnboardingBackendConnected() call should be sufficient — just verifying the pattern is consistent.

2. Snapshot file naming: The new snapshot onboarding-step-0-check-backend.png follows the existing naming convention. Good consistency.

3. Timeout values: All timeout values (10s, 15s) are reasonable for E2E tests running in CI environments.

---

Test Plan Verification

The PR includes the following test commands:

npm run typecheck
npx eslint tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts tests/e2e/snapshots/onboarding.snapshot.spec.ts
MOCK_LLM_SESSION_API_KEY=mock-llm-onboarding-key MOCK_LLM_PYTHON=.mock-llm-venv/bin/python3 npx playwright test --config=playwright.mock-llm.config.ts tests/e2e/mock-llm/mock-llm-onboarding-regressions.spec.ts
npx playwright test tests/e2e/snapshots/onboarding.snapshot.spec.ts --project=chromium --update-snapshots --reporter=list

These commands are comprehensive and cover the new tests and updated snapshots.

---

Conclusion

This PR is well-structured and adds valuable regression coverage. The tests are clear, well-documented, and follow the project's testing patterns. No blocking issues identified.

Recommendation: Approve

---

This review was generated by an AI agent (OpenHands) on behalf of the user through OpenHands Automation. View conversation

[openhands-release-bot]

🚀 Released in v1.2.0.

[github-actions]

📸 Snapshot Test Report

Warning

Snapshot comparison step crashed (timeout, OOM, or runner error) — diff results below may be incomplete or absent.
Check the CI logs for the full error output (look for the "Run snapshot comparison" step).

✅ No baseline found on main — all 73 snapshots are new and will become the baseline once this PR merges.

Category	Count
🔴 Changed	0
🆕 New	73
✅ Unchanged	0
Total	73

🆕 New snapshots (73)

These snapshots have no baseline on main and will become the new baseline once this PR merges.

archived-conversation — 3 snapshots

conversation-panel-with-archived-badges

conversation-view-archived

conversation-view-sandbox-error

automations — 4 snapshots

automations-delete-modal

automations-list-active-inactive

automations-no-automations

automations-search-no-results

backends-extended — 19 snapshots

backend-add-blank-disabled

backend-add-cloud-advanced-open

backend-add-cloud-no-key-disabled

backend-add-cloud-with-key-enabled

backend-add-form-partially-filled

backend-add-invalid-url-disabled

backend-add-local-ready

backend-add-name-only-disabled

backend-add-two-column-layout

backend-add-whitespace-host-disabled

backend-after-switch

backend-cancel-nothing-saved

backend-dropdown-two-backends

backend-edit-prefilled

backend-manage-after-removal

backend-manage-two-listed

backend-remove-cancelled

backend-remove-confirmation

backend-switch-overlay

backends — 3 snapshots

backend-add-modal

backend-manage-modal

backend-selector-open

changes-tab — 3 snapshots

changes-deleted-file

changes-diff-viewer

changes-empty

collapsible-thinking — 4 snapshots

reasoning-content-collapsed

reasoning-content-expanded

think-action-collapsed

think-action-expanded

mcp-page — 11 snapshots

mcp-custom-server-1-editor-open

mcp-custom-server-2-url-filled

mcp-custom-server-3-all-filled

mcp-custom-server-4-installed

mcp-custom-server-editor

mcp-empty-installed

mcp-search-filtered

mcp-slack-install-1-marketplace

mcp-slack-install-2-modal

mcp-slack-install-3-filled

mcp-slack-install-4-installed

onboarding — 4 snapshots

onboarding-step-0-check-backend

onboarding-step-1-choose-agent

onboarding-step-2-setup-llm

onboarding-step-3-say-hello

projects-workspace-browser

projects-workspace-browser

settings-page — 5 snapshots

add-backend-modal

analytics-consent-modal

home-screen

settings-app-page

settings-page

settings-secrets — 5 snapshots

secrets-add-form-filled

secrets-add-form

secrets-after-save

secrets-delete-confirm

secrets-list

settings-verification — 3 snapshots

condenser-settings

verification-settings-off

verification-settings-on

sidebar — 3 snapshots

sidebar-collapsed

sidebar-conversation-panel

sidebar-filter-menu

skills-page — 5 snapshots

skills-empty

skills-loaded

skills-no-match

skills-search-filtered

skills-type-filter

---

Generated by the Snapshot Tests workflow. This comment was created by an AI agent (OpenHands) on behalf of the repo maintainers.