feat(app): Browser Automations — smart connect + UI redesign (WIP)#3416
Draft
tofikwest wants to merge 161 commits into
Draft
feat(app): Browser Automations — smart connect + UI redesign (WIP)#3416tofikwest wants to merge 161 commits into
tofikwest wants to merge 161 commits into
Conversation
Adds the first step of the browser-automation connect wizard: a credentials form (username, password, optional authenticator setup key) shown before the live sign-in. On submit it resolves the auth profile and stores the login via the credentials endpoint so scheduled and manual runs can sign in on their own. Credential storage failures are non-fatal and fall back to manual sign-in. Part of the Browser Automations redesign (connect-a-login flow). Adds unit tests for the form (fields, validation, submit, password toggle).
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Replaces the bare "enter a URL" prompt with the three-step explainer (01 Connect a login / 02 Describe what to capture / 03 Evidence on schedule), a single "Connect a vendor login" action, and the 1Password reassurance line. The URL is now collected in the connect credentials form rather than here. Part of the Browser Automations redesign (first-time setup).
Switches the first-time setup screen to the designer's 2c variant — a split card with the pitch and "Connect a Vendor Login" action on the left and a quiet "How it works" rail (01/02/03) on the right. Stacks to one column on mobile.
Adds a login-analysis backend: POST /v1/browserbase/analyze-login opens the vendor sign-in page in a throwaway cloud browser, detects the supported login methods (password / SSO / passkey, identifier type, extra fields), and returns a recommendation (ready / works-with-check-ins / manual). Reads a public page only — no credentials — and always degrades to a manual fallback if it can't be read. Foundation for the smart-connect flow. Unit tests for the recommendation logic and the analyzer service.
Replaces the connect flow with the "Browser-First Split" (1C) design: a stepper rail plus a browser stage that opens the vendor URL, runs AI login-method detection (/analyze-login), shows a recommendation (ready / works-with-check-ins / manual), opens the live sign-in, then captures the reusable credentials (username, password, authenticator setup key) after verification. Removes the superseded credentials-first form. Adds tests for the capture form. The SMS-only -> "enable an authenticator app" recommendation is deferred to the live sign-in step (2FA method isn't visible before sign-in).
tofikwest
changed the title
Rewrites the automations list to group automations under their vendor connection. Each connection shows its status (Connected / Needs reconnect / Needs your action / Not connected) with a one-click Reconnect for the ones that need a human. Adds a useBrowserProfiles hook to load connection statuses and wires reconnect through the existing live sign-in.
Extends credential capture beyond username/password/TOTP to any site-specific fields a login needs. The capture form gains an "add a field" control; the API stores extras as custom 1Password fields, resolves them at run time (via items.get, additive to the existing resolve path so the standard path is unchanged), and the sign-in agent fills each by label using variable substitution (only labels reach the model, never values).
…lable Move Browserbase session/context creation inside the try/catch so a missing BROWSERBASE_API_KEY (or any Browserbase outage) returns the manual-entry fallback instead of a 500 at the analyze step. Documents the required BROWSERBASE_API_KEY / BROWSERBASE_PROJECT_ID in .env.example.
… regex Replace the hardcoded sign-in URL patterns (looksLikeSignInUrl) with a URL-aware classifier: pass the browser's current URL into the login classifier so the model itself decides whether we're on the real app or still on a sign-in / identity-provider page. It generalizes to any vendor (it already knows hosts like signin.aws.amazon.com or login.microsoftonline.com) without us enumerating patterns, and still fixes the AWS false "signed in" — the classifier no longer calls a mid-redirect / sign-in page "logged in". The URL is a hint layered on the page content; it degrades to content-only when unavailable.
navigateToSignIn opened the "Sign in" link then waited a flat 1.5s — too short for vendors that bounce through a redirect chain (e.g. AWS from the marketing homepage → signin.aws.amazon.com), so classification and credential fill ran against the stale homepage (no fields), producing the misleading "Entering username/password" steps on a page that has none. Now wait until the URL stops changing (bounded ~5.6s) before proceeding, so we act on the real sign-in form. Also nudge the instruction to include "Sign in to Console"-style links. (For AWS, pasting the direct sign-in URL still avoids the homepage hunt entirely.)
…teps The live sign-in steps showed a generic "Entering username" even when the vendor's field is something else (e.g. AWS "IAM username") — extra fields already stream their detected labels, but the identifier didn't, which read as hardcoded. Thread the detected identifier label from the capture form through the sign-in call to the step log, so the streamed step names the real field. Kept out-of-band from credential storage/resolution (no change to how secrets are stored) and the fill stays semantic, so nothing about the working sign-in changes; scheduled runs fall back to the generic label. Password label is universal, left as-is.
Cancelling the connect flow left its resume-state in localStorage, so re-opening "Connect a vendor" reloaded the last in-progress step (e.g. "choose sign-in method" for the previous vendor) with no way back — the user couldn't start a new vendor or escape. handleCancel now clears the resume-state (fixes both the Connections page and the evidence task, since the flow is shared). The Connections page additionally clears on open, so "Connect a vendor" there always starts fresh at URL entry (a settings page has no "resume where I left off").
When a vendor opens its sign-in in a new tab (e.g. AWS from its marketing homepage), the AI signed in on the real form — reaching the 2FA prompt — but the session live view kept showing the front tab (the homepage), so the user couldn't see the sign-in page or complete the 2FA take-over. Evidence runs already handle this via resolveEvidencePage + bringEvidencePageToFront; the connect sign-in never did. Reuse resolveEvidencePage after navigating to the sign-in form to bring the right page to front (best-effort). For AWS, pasting the direct sign-in URL also reduces the redirect/tab hopping.
resolveEvidencePage prefers the initial page when the target hostname doesn't match (AWS's entered URL redirects to a different host), so it brought the marketing homepage to front instead of the sign-in tab — making the live-view mismatch worse. Revert to reassess with a correct multi-tab approach.
When a vendor opens its sign-in in a new tab (e.g. AWS from its homepage), the AI worked on that tab but the live view stayed pinned to the original (the marketing homepage), so the user couldn't watch — or complete a 2FA take-over on — the real form. Follow the active tab (designer option B): after navigating to the sign-in (and after the SSO click), resolve the live-view URL of the tab the AI is actually on via Browserbase `sessions.debug()` per-tab URLs, and stream it over the run's metadata as `signinLiveViewUrl`; the connect flow re-points its iframe to it. Tab selection (match by URL, then hostname, then newest) is a pure `selectLiveViewUrl` with unit tests. Best-effort — the live view keeps its current tab if lookup fails.
…page An automation's screenshot and its pass/fail check disagreed: the check read the real page (via stagehand) — e.g. the AWS IAM users list — but the screenshot captured the marketing homepage. Cause: selectEvidencePage matched the ENTERED URL's host (the sign-in host), and when the agent navigated to a different host (console.aws.amazon.com) in a new tab, no tab matched, so it fell back to the initial (homepage) tab. Prefer the newest non-initial tab (where the agent actually ended up) over the stale initial page, so the screenshot matches what the check judged. Same-tab navigations are unchanged (only the initial page is open).
Users must give Comp AI the authenticator setup key (TOTP seed), not the 6-digit code, for unattended scheduled sign-ins — but they rarely know where to find it. Add AI-generated, per-vendor instructions (no hardcode): - BrowserMfaInstructionsService: generateObject (Sonnet) + strict schema with a confidence flag; confidence-gated to a single universal fallback so we never show shaky invented steps. In-memory per-hostname TTL cache (1 day) — public, cheap-to-regenerate guidance, so no DB migration; a shared DB cache is a drop-in swap behind getInstructions. Web-search grounding is the next step. - GET /v1/browserbase/mfa-instructions?host=... (integration:read). - MfaSetupHelp: lazy 'How do I find this key?' collapsible under the setup-key field, shown in both the task and settings connect flows. - Manage panel 'add 2FA later': 2FA checkbox + setup-key field + helper in Change login, reusing the credentials endpoint that already stores the TOTP seed. Tests: api 4, app 6 (+ existing browser suites green: api 140, app 97).
…e docs Before generating the per-vendor authenticator instructions, pull the vendor's current help docs via Firecrawl web search (v2/search) and feed them to the model, so steps track the live UI instead of the model's training snapshot. - Best-effort: no FIRECRAWL_API_KEY, a failed/timed-out request, or empty results all fall back to ungrounded generation — grounding never blocks, and the confidence gate still protects the output. - Grounding runs inside generate(), so the per-hostname cache means Firecrawl is hit at most once per vendor per day. - New 'grounded' flag flows to the UI: MfaSetupHelp shows a subtle 'Checked against this vendor's current help docs' trust line when true. Tests: api +2 (grounded path, Firecrawl-failure path), app +1 (trust line). Suites green: api 142, app 98.
Surface 1 (2a — quiet disclosure): the 'How do I find this key?' helper is now a muted link-row that opens a soft panel with numbered per-vendor steps and a single footer line — a green 'Checked against <vendor>'s current help docs · <date>' trust line when grounded, or an honest tinted generic-fallback note. Loading and error (with retry) states included. Adds a checkedAt timestamp to the API payload. Surface 2 (2d — Automatic 2FA row): 'add 2FA later' moves out of Change login into its own row in the Manage panel's DETAILS group with an ON / NOT SET UP pill. 'Add authenticator key' stores ONLY the TOTP seed — no username/password re-entry: - New credential-storage methods read/attach/remove just the vault item's TOTP field (1Password items.get + put); status is read live from the vault, so no DB flag can drift and no migration is needed. - Endpoints: GET/POST/DELETE /v1/browserbase/profiles/:id/totp (integration read/update). useTotpStatus hook drives the pill; Replace / Turn off when on. Tests: api +7 (storage TOTP status/set/clear), app 2FA-row + 2a states green. Suites: api 149, app 100.
…idance - The model sometimes returns markdown bold (**Settings**); we render raw text, so the asterisks showed literally. Strip **/__/backticks in the service and tell the model to write plain text — the 2a design shows unstyled steps. - Prefetch the per-vendor guidance while the user fills in the connect form, so the 'How do I find this key?' helper opens instantly. Bounded by the per-host cache, so this just front-loads the once-per-vendor-per-day generation and warms it for the next person connecting the same vendor. Tests: api +1 (markdown strip). Suites: api 150, app 100.
…hint - Native <button>s (header schedule / Connect another vendor / New evidence, the MfaSetupHelp trigger + retry, Turn off, Add-a-field) had no cursor:pointer. - The 2FA setup-key hint dropped the spaces around <em>once</em> / <strong>not</strong> because the tags sat on line breaks (JSX strips whitespace adjacent to tags), rendering 'oncewhen' / 'notthe'. Rewrote it as plain text (matches the 2a design).
During 'Test this step', the agent opens/switches tabs (a vendor may open sign-in or a console in a new tab), but the composer's live view stayed on the starting tab. Poll the newest tab during the agent run and stream its live-view URL (metadata 'testLiveViewUrl'); the composer points the iframe at it. Same tab-following we already do for the sign-in flow. Best-effort — a follower error never affects the run. Mirrors the evidence-screenshot fix so watch matches result.
… reconnect On reconnect (and scheduled relogin) the sign-in step showed a generic 'Entering username' because the detected field label (e.g. 'IAM username') was only passed transiently from the connect capture form. Persist it on the connection at connect time and fall back to it at sign-in when the caller doesn't supply one, so the real field name shows uniformly. - New BrowserAuthProfile.identifierLabel column (+ migration). - Capture flow stores it via POST /credentials (usernameLabel); sign-in uses input.usernameLabel ?? profile.identifierLabel. Tests: api +1 (label persisted, trimmed). Suites: api 150, app 100. NOTE: run 'bunx prisma migrate deploy' (packages/db) on the stand before deploy.
…idence section The 'Drafts — not saved yet' strip rendered as a standalone card above the section, so it read as unattached — unclear it belongs to Browser evidence. Move it inside the section card, as an inset band directly under the header (new DraftsStrip 'nested' variant). The standalone card is kept only for the no-automations-yet state, where there's no section to nest into.
…ons' 'Connections' was ambiguous (easily confused with integration connections). Name it 'Browser connections' so it's clear it manages the vendor logins browser automations sign in with. Label only — the /settings/browser-connection route is unchanged. Updated the sidebar item, the browser-tab title, and the active-tab label.
Someone landing on the page from Settings had no idea what it's for. Wrap it in a
titled Section ('Browser connections' + one-line description) and add a compact
'How it works' explainer: (1) connect a vendor login here — Comp signs in incl.
2FA, (2) it signs in on a schedule, screenshots audit evidence, and re-signs in on
its own when a session expires, (3) the automations that use these logins are
built inside a task under 'Browser evidence' — so a lost user knows where to go.
- The Section repeated 'Browser connections' under the layout's big page title — removed the Section title, kept the description + Connect action. - 'How it works' steps now use a flex row (number + text with a gap) so the spacing after the number is consistent — item 3 was rendering as '3.You'.
Tie the copy to the product's real terms so people know where these logins apply: description now reads 'The vendor logins your evidence tasks use to sign in and capture audit evidence automatically', and step 3 points to adding automations inside an evidence task's Browser evidence section.
…below the table - Add a search box above the table (filters by vendor name, hostname, or login). - Paginate at 10 rows/page with Previous / Page X of Y / Next (shown only when a filtered list exceeds one page) — previously every connection rendered at once. - Move the 'N connections · M active' summary from above to below the table, next to the pager. Empty-search shows a 'No connections match …' row.
…ount reflect it - Search now lives in a toolbar at the top of the table card (right-aligned on desktop, full-width on mobile) with a magnifier icon, instead of floating above. - The summary under the table now reflects the search: 'X of Y connections · N active' while filtering (active/attention counts are of the shown set), and the plain total when not searching.
…ide) Move the search out of a separate toolbar and into the column-header row itself — right-aligned on the same line as the VENDOR/… headers (desktop). On mobile the table scrolls horizontally, so the search stays as a full-width bar above the table instead of hiding behind the scroll. Both share one query state.
Put the search right of the 'Vendor' label (left of 'Method'), inside the Vendor header cell, instead of as its own right-side column. Mobile keeps the full-width search bar above the table.
Mintlify page (packages/docs/browser-automations.mdx) covering what the feature does, connecting a vendor, unattended 2FA via the authenticator setup key, creating a browser-evidence automation, what happens on a schedule (self-healing re-login), managing connections, best practices, and troubleshooting. Added to the Guides → Get Started nav next to Automated Evidence.
A few plain-language bullets on where and why it's most useful — reaching evidence with no API, handling 2FA unattended, staying fresh on its own, and being auditor-ready — plus the sweet-spot use case (recurring evidence from admin consoles / security-settings pages that have no export).
Four screenshots, one per key moment: the audit-ready evidence output (hero), the 2FA setup-key step with per-vendor guidance (Unattended 2FA), the live test run with the AI driving the browser (Create an automation), and the Browser connections page (Manage your connections).
… hero Swap the hero for a version with the org's repo navigation blurred out — keeps the positive, secret-free part (2FA enforced, source URL) for the public docs.
Make explicit that automations run automatically on a schedule you control (Daily / Weekly / Monthly / Quarterly / Yearly, one cadence per task, changeable any time) and that you can also Run any automation on demand — not only manually.
…ndpoints The raw session/context operations (create, close, navigate, check-auth, and the client-supplied session in execute / execute-live) acted on an id taken from the request without confirming it belongs to the caller's organization. Add tenant ownership checks — assertContextOwnedByOrg / assertSessionOwnedByOrg — so these operations only ever touch the caller's own browser contexts and sessions. The close path stays tolerant (a session that can't be resolved is a no-op). Tests: +6 tenant-guard cases. Browserbase suite green (157).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Feature is functionally code-complete end-to-end
POST /analyze-login: reads a vendor's login methods, returns a recommendation. Public page only, no creds, SSRF-guarded, manual fallback.Verification
Not done — needs you / infra (can't be done from code alone)
OP_SERVICE_ACCOUNT_TOKEN, PostHog flag on, a real Browserbase run, a deploy. This also confirms the 1Password WASM loads in the Trigger worker.Cosmetic (already covered, not separate components)
sm).