[Feat] Add a private browser to Fast Sessions with the browse tool - #2420
[Feat] Add a private browser to Fast Sessions with the browse tool#2420mrubens wants to merge 1 commit into
Conversation
Fast Sessions get a browse tool backed by agent-browser running against a Browser Use cloud browser from the api process, so a Session can open pages, read them, act on them, and capture screenshots or recordings without a sandbox. Commands are tokenized without a shell, subcommands are allowlisted, and flags that could reach another session, a host binary, a profile, or a different provider are refused. Captures upload as Session-owned visual-proof artifacts (a binary sibling of the text artifact path) with a signed raw URL, render inline in the web transcript, and can ride on Slack replies: the image and video resolvers now accept Session-owned artifacts by mapping Fast conversation ids to their unified session rows. GPT-family models treat a screenshot as delivered by their final message and do not call send_chat_reply afterwards, so the delivery decision lives on the capture itself as deliverToUser; the next visible reply carries what the model opted into. The tool is hidden unless R_FAST_BROWSER_PROVIDER names a provider. The api image installs the agent-browser CLI; no Chrome ships in it.
|
2 issues outstanding. See task
Reviewed 7b1a4a3 |
![roomote-community[bot]](https://avatars.githubusercontent.com/u/211522643?s=60&v=4){kind=small}
| 'eval', | ||
| 'tab', | ||
| 'set', | ||
| 'cookies', |
There was a problem hiding this comment.
cookies is allowlisted here, and validateBrowseCommand does not reject cookies set --curl <path>. In agent-browser 0.37.1, that form imports cookie data by reading the supplied host file, so a Fast model can make the API-side CLI read arbitrary files despite the intended host-filesystem boundary. Remove this file-backed form (or reject --curl and any other path-bearing cookie variants) before exposing the tool.
| !reply.imageArtifactIds?.length && defaultImageArtifactIds.length | ||
| ? { ...reply, imageArtifactIds: defaultImageArtifactIds } | ||
| : reply; | ||
| const replyWithImages = attachRequestedCaptures(reply); |
There was a problem hiding this comment.
When an inference retry notice is being replaced, this sends the capture-bearing reply through replaceInferenceRetryReply. The surface replacers only consume message (for example, the Slack replacer never resolves imageArtifactIds or delivers videoArtifactIds), then return true, so the normal delivery path is skipped. Because attachRequestedCaptures has already spliced the pending IDs, a deliverToUser browse capture taken after a retry produces an edited text reply with no media. Bypass replacement for attachment-bearing replies or add attachment delivery to every replacer.
1 participant
What
Fast Sessions get a
browsetool: a private browser per conversation, with no sandbox. The api process runs the agent-browser CLI against a Browser Use cloud browser, so a Session can open a page, read it, act on it, and capture screenshots or recordings for QA, error lookups, and other browser-only work that previously needed a coding task.How it works
open,snapshot -i,click @e3,fill,press,screenshot,record start|stop,close, …). Commands are tokenized without a shell, the subcommand is allowlisted, and flags that could reach another session, attach to a CDP endpoint, launch a host binary, load a profile or extension, or switch providers are refused. The child process gets a minimal environment with only the provider key.screenshotandrecord stopupload as Session-ownedvisual-proofartifacts through a new binary sibling of the text artifact path (createSessionMediaArtifact, 50 MiB cap, signed raw URL). They render inline in the web transcript and in the Session Artifacts panel.send_chat_reply.deliverToUser. GPT-family models treat a screenshot as delivered by their final message and do not callsend_chat_replyafterwards (see below), so the delivery decision lives on the capture call itself. The harness attaches only the captures the model opted into, on its next reply.R_FAST_BROWSER_PROVIDERthe tool is filtered out of the model's catalog rather than failing per call.Configuration
R_FAST_BROWSER_PROVIDERbrowseruseto enable;locallaunches Chrome on the api host (development only).R_BROWSER_USE_API_KEYR_AGENT_BROWSER_PATHPATH.All three are on the control-plane env denylist. The app image installs agent-browser globally; no Chrome ships in it.
Why
deliverToUserinstead of animageArtifactIdsinstructionReplaying the exact decision point via OpenRouter with the real tool descriptions and a real screenshot result:
deliverToUseron the screenshot callIn every GPT failure the model made no tool call and ended its turn with "Here's a screenshot of …", expecting the harness to render the image. For GPT-family models, anything that must happen after a tool belongs on the tool.
Tests
browsecaptures.Known gaps
localprovider path is typechecked and unit-tested but has not been run.eval,cookies, andstorageare allowed; worth revisiting before users sign the browser into anything.