|
1 | 1 | --- |
2 | 2 | name: browser-execute |
3 | | -description: Use ONLY when calling the `browser_execute` tool or driving a real browser via the Chrome DevTools Protocol. Required reading before the first `browser_execute` call in a session. Covers the three connection methods (local Chrome with remote debugging, isolated debug-port profile, Browser Use cloud), the in-process `session` / `console` snippet model, attaching to a page target, common CDP commands, the per-project `.bcode/agent-workspace/` for reusable scripts, and screenshot auto-attachment. |
| 3 | +description: Required reference for driving a browser with browser_execute and its persistent CDP session. |
4 | 4 | --- |
5 | 5 |
|
6 | | -The `browser_execute` tool evaluates JavaScript against a connected browser `session` via the Chrome DevTools Protocol. |
7 | | -The snippet runs in-process; `session` is bound to a long-lived CDP `Session` that persists. |
8 | | -There is no helper namespace, just `session`, `console`, and standard JS globals. |
| 6 | +`browser_execute` runs JavaScript with `session`, `console`, and standard JS globals. The CDP `session` persists across |
| 7 | +calls, but local JavaScript variables do not. Use short deterministic snippets, print or return compact structured |
| 8 | +results, and checkpoint large collections under `./.bcode/agent-workspace/`. |
9 | 9 |
|
10 | | -Workspace: `<projectRoot>/.bcode/agent-workspace/`. Read/write your reusable scripts here. |
11 | | -Skills: `{{SKILLS_DIR}}/`. Read-only browser execute reference docs. |
| 10 | +This read-only skill is materialized under `{{SKILLS_DIR}}/browser-execute/` for the current run. |
12 | 11 |
|
13 | | -## Connecting |
14 | | -In Browser Use Cloud API V4, `browser_execute` automatically connects and attaches the existing page once when the fresh run first uses this tool; do not call `session.connect()` or `session.use()` before driving it. |
15 | | -Otherwise, call `session.connect(...)` once at the start of your work. There are three connection methods: |
| 12 | +## Connect |
16 | 13 |
|
17 | | -#### Way 1: connect to the user's running Chrome or Chromium-based browser (real profile, popup-gated). |
18 | | -Choose when the task involves the user's logged-in sites, current browser state, cookies, saved data, etc. |
| 14 | +Browser Use Cloud API V4 automatically connects and attaches the existing page. When both `V4_RUN_ID` and |
| 15 | +`BU_CDP_WS` or `BU_CDP_URL` are set, start driving immediately; do not call `session.connect()` or `session.use()`. |
19 | 16 |
|
20 | | -```js |
21 | | -// Attempts to connect to every detected Chrome, most-recently-launched first. |
22 | | -await session.connect() |
23 | | -``` |
24 | | - |
25 | | -For this to work the user must have navigated to `chrome://inspect/#remote-debugging` in their target Chrome and ticked "Allow remote debugging for this browser instance". This setting is per-profile and persists across every future launch of that profile. On Chrome 144 and later, the first attach also triggers an in-browser "Allow remote debugging?" popup that the user must click "Allow" on. The popup may reappear on later attaches under conditions that are not fully characterized — browser restart, time elapsed, new CDP session. Ask the user to click Allow again if a previously working connection starts 403'ing. |
26 | | - |
27 | | -Failure modes: |
28 | | -- `connect()` throws "No running browser with remote debugging detected". The checkbox at `chrome://inspect/#remote-debugging` has not been ticked in any running Chrome profile, or no Chrome is running. |
29 | | -- `connect()` throws with "403" / "permission" / "WS closed before open". The checkbox is ticked but the user hasn't clicked Allow on the popup yet. By default `connect()` errors in 5s; pass `{ timeoutMs: 30000 }` to wait up to 30s for the click. |
30 | | - |
31 | | -#### Way 2: connect to a Chrome or Chromium-based browser launched with a debug port (isolated profile, no popups). |
32 | | -Choose for unattended automation, or for an isolated browser. |
33 | | - |
34 | | -Launch Chrome with `--remote-debugging-port=<port> --user-data-dir=<path>`. Pick a directory you can access — e.g., a project-local one like `./.bcode/chrome-data-dir`. |
35 | | - |
36 | | -```bash |
37 | | -# Linux |
38 | | -google-chrome --remote-debugging-port=9222 --user-data-dir=./.bcode/chrome-data-dir |
39 | | -# macOS |
40 | | -"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \ |
41 | | - --remote-debugging-port=9222 --user-data-dir=./.bcode/chrome-data-dir |
42 | | -# Windows (cmd.exe) |
43 | | -"C:\Program Files\Google\Chrome\Application\chrome.exe" ^ |
44 | | - --remote-debugging-port=9222 --user-data-dir=.\.bcode\chrome-data-dir |
45 | | -# Windows (PowerShell) |
46 | | -& "C:\Program Files\Google\Chrome\Application\chrome.exe" ` |
47 | | - --remote-debugging-port=9222 --user-data-dir=.\.bcode\chrome-data-dir |
48 | | -``` |
49 | | - |
50 | | -```js |
51 | | -// Resolve the live WebSocket URL via `/json/version` and connect: |
52 | | -const ver = await fetch("http://127.0.0.1:9222/json/version").then(r => r.json()) |
53 | | -await session.connect({ wsUrl: ver.webSocketDebuggerUrl }) |
54 | | -``` |
55 | | -
|
56 | | -`--user-data-dir` must not be Chrome's platform default. Chrome 136 and later silently no-ops the `--remote-debugging-port` flag when `--user-data-dir` is the platform default. The platform defaults are `%LOCALAPPDATA%\Google\Chrome\User Data` on Windows, `~/Library/Application Support/Google/Chrome` on macOS, `~/.config/google-chrome` on Linux. |
57 | | -You cannot reuse the user's everyday Chrome profile by copying its files into a custom directory. |
58 | | -
|
59 | | -Failure modes: |
60 | | -- Chrome's launch log prints `DevTools listening on ws://...:<port>/...` immediately followed by `bind() failed: Address already in use` and Chrome exits. Confirm the port is actually open with `curl http://127.0.0.1:<port>/json/version` before connecting. |
61 | | -- `{ profileDir }` raises ENOENT on `DevToolsActivePort`. Chrome 147+ doesn't write this file under custom `--user-data-dir`; use the `/json/version` route above instead. |
62 | | -- Launch silently no-ops `--remote-debugging-port`. Launching a second Chrome that points at a `--user-data-dir` matching a running process ignores `--remote-debugging-port`. |
63 | | -
|
64 | | -#### Way 3: provision and connect to a Browser Use cloud browser. |
65 | | -Choose when the computer has no browser or to work in the background. |
66 | | -
|
67 | | -```js |
68 | | -// Provision and connect to a cloud browser |
69 | | -const r = await fetch("https://api.browser-use.com/api/v3/browsers", { |
70 | | - method: "POST", |
71 | | - headers: { "X-Browser-Use-API-Key": process.env.BROWSER_USE_API_KEY, "Content-Type": "application/json" }, |
72 | | - body: "{}", |
73 | | -}) |
74 | | -// Additional options: fetch https://docs.browser-use.com/cloud/api-v3/browsers/create-browser-session |
75 | | -const { id, cdpUrl, liveUrl } = await r.json() |
76 | | -const ver = await fetch(`${cdpUrl}/json/version`).then(r => r.json()) |
77 | | -await session.connect({ wsUrl: ver.webSocketDebuggerUrl }) |
78 | | -console.log("liveUrl for the user to watch:", liveUrl) |
79 | | -``` |
80 | | -
|
81 | | -```js |
82 | | -// Browser Use cloud will eventually close idle browsers. An explicit stop frees the slot: |
83 | | -await fetch(`https://api.browser-use.com/api/v3/browsers/${id}`, { |
84 | | - method: "PATCH", |
85 | | - headers: { "X-Browser-Use-API-Key": process.env.BROWSER_USE_API_KEY, "Content-Type": "application/json" }, |
86 | | - body: JSON.stringify({ action: "stop" }), |
87 | | -}) |
88 | | -``` |
89 | | -
|
90 | | -A cloud browser is a paid service and requires `BROWSER_USE_API_KEY` to be set in the environment. |
91 | | -Browser Use cloud browsers auto-solve captchas. When you land on one, just stop driving and `await new Promise(r => setTimeout(r, 10000))`; the solver runs server-side and the page advances on its own. |
92 | | -Browser Use has a free tier gated for intelligent and powerful agents. Unlimited free browsers and proxies, limited to 3 concurrent browsers. Agents can sign up and receive a key fully autonomously, but must pass a proof-of-agent-capability challenge. In can be done without any browser. To begin, post or webfetch https://browser-use.com and solve the challenge (see https://browser-use.com/llms.txt or POST https://api.browser-use.com/cloud/signup.). |
93 | | -
|
94 | | -#### Way 4: user-preconfigured endpoint |
95 | | -Not a method you choose — a way for the user to hand you a pre-set endpoint. |
96 | | -When `V4_RUN_ID` and `BU_CDP_WS` (or its alias `BU_CDP_URL`) are both set, `browser_execute` connects to that endpoint and attaches its existing non-internal page once before the first snippet. Go straight to driving it. Other environments keep the explicit connection flow, and explicit `{ wsUrl }` / `{ profileDir }` calls still connect to the requested endpoint instead. |
97 | | -If that fixed endpoint closes or repeatedly fails its WebSocket upgrade, reconnecting to the same URL cannot recover it; the endpoint owner must replace it. |
98 | | -
|
99 | | -## Attaching to a target |
100 | | -After connecting manually, attach to a page target before driving the browser. A preconfigured endpoint is already attached automatically: |
| 17 | +Otherwise connect once, then attach a non-internal page: |
101 | 18 |
|
102 | 19 | ```js |
| 20 | +await session.connect() |
103 | 21 | const targets = (await session.Target.getTargets({})).targetInfos |
104 | | -// Pick the first non-internal tab if none was specified. |
105 | 22 | const page = targets.find(t => t.type === "page" && !t.url.startsWith("chrome://")) |
106 | 23 | await session.use(page.targetId) |
107 | 24 | ``` |
108 | 25 |
|
109 | | -If a target-scoped command throws `CdpError` code `-32001` (`Session with given id not found`), the browser connection is still usable but the target session is stale. List targets again, `session.use(...)` the intended page, and retry the rejected command once. Calling `session.connect()` without arguments is a no-op while connected; it does not replace a stale target session. |
| 26 | +An explicit `{wsUrl}` connects to a chosen CDP endpoint. Every reconnect or browser switch clears the target attachment, |
| 27 | +so list targets and call `session.use(...)` again. If a target command throws `-32001`, reattach the intended page and |
| 28 | +retry once. Opening a tab does not switch the attachment. |
110 | 29 |
|
111 | | -Every explicit reconnect or browser switch retires the previous socket and clears its active target attachment. Re-list targets, call `session.use(...)`, and rediscover DOM nodes and Runtime objects before continuing. |
| 30 | +## Drive |
112 | 31 |
|
113 | | -Opening a tab creates a new `page` target but does not switch the active attachment. Call `Target.getTargets` again and `session.use(targetId)` when continuing there. |
114 | | -
|
115 | | -## Driving a page |
116 | | -Domain methods follow `session.<Domain>.<method>(params)` and return Promises. |
117 | | -The full surface (652 commands) is the Chrome DevTools Protocol. |
118 | | -`Object.keys(session.domains).sort()` lists every CDP domain bound on the session; `Object.keys(session.Page).sort()` lists the methods for `Page`. |
119 | | -For unknown param shapes, call with `{}` and inspect the thrown `CdpError` — `.data` carries the missing-field detail. |
120 | | -
|
121 | | -Common moves: |
| 32 | +CDP methods are available as `session.<Domain>.<method>(params)`. Inspect the surface with |
| 33 | +`Object.keys(session.domains).sort()` or `Object.keys(session.Page).sort()`. |
122 | 34 |
|
123 | 35 | ```js |
124 | | -// Navigate. Register the load waiter BEFORE navigate so a fast load isn't missed. |
125 | 36 | await session.Page.enable() |
126 | | -const loaded = session.waitFor("Page.loadEventFired", { timeoutMs: 15_000 }) |
127 | | -await session.Page.navigate({ url: "https://example.com" }) |
| 37 | +const loaded = session.waitFor("Page.loadEventFired", {timeoutMs: 15_000}) |
| 38 | +const navigation = await session.Page.navigate({url: "https://example.com"}) |
128 | 39 | await loaded |
129 | | -// Page.navigate resolves even on network errors — its result carries `errorText` when the load failed. |
| 40 | +if (navigation.errorText) throw new Error(navigation.errorText) |
130 | 41 |
|
131 | | -// Evaluate JS in the page. |
132 | | -const r = await session.Runtime.evaluate({ |
133 | | - expression: "document.title", |
| 42 | +const result = await session.Runtime.evaluate({ |
| 43 | + expression: `JSON.stringify({title: document.title, text: document.body.innerText.slice(0, 4000)})`, |
134 | 44 | returnByValue: true, |
135 | 45 | }) |
136 | | -console.log(r.result.value) |
137 | | -
|
138 | | -// Click by coordinates. |
139 | | -const x = 200, y = 300 |
140 | | -await session.Input.dispatchMouseEvent({ type: "mouseMoved", x, y }) |
141 | | -await session.Input.dispatchMouseEvent({ type: "mousePressed", x, y, button: "left", clickCount: 1 }) |
142 | | -await session.Input.dispatchMouseEvent({ type: "mouseReleased", x, y, button: "left", clickCount: 1 }) |
143 | | -
|
144 | | -// Type text. |
145 | | -await session.Input.insertText({ text: "hello" }) |
146 | | -
|
147 | | -// Screenshot. |
148 | | -await session.Page.captureScreenshot({ format: "png" }) |
149 | | -// You see the image inline on the next turn — `browser_execute` automatically |
150 | | -// attaches every `Page.captureScreenshot` result. No need to decode, save, or |
151 | | -// `read` the bytes back. The base64 is still in `data` (via the return value) |
152 | | -// for the rare case you want to process it programmatically. |
153 | | -``` |
154 | | -
|
155 | | -`Page.navigate` can return a non-empty `errorText` instead of throwing. Treat it as a failed navigation. If `ERR_TUNNEL_CONNECTION_FAILED` persists, reloading, reattaching, or reconnecting to the same endpoint cannot change its proxy route; use another source or replace the cloud browser instead of retrying it. |
| 46 | +console.log(result.result.value) |
156 | 47 |
|
157 | | -Not auto-attaching a screenshot: attachment is keyed on the `Page.captureScreenshot` response; pixels that arrive as an event are never attached. |
| 48 | +await session.Runtime.evaluate({ |
| 49 | + expression: `document.querySelector("button")?.click()`, |
| 50 | + returnByValue: true, |
| 51 | +}) |
158 | 52 |
|
159 | | -```js |
160 | | -await session.Page.enable() |
161 | | -// Optional: screencast has no `clip`, so override the viewport for exact frame size. |
162 | | -await session.Emulation.setDeviceMetricsOverride({ width: 1200, height: 630, deviceScaleFactor: 1, mobile: false }) |
163 | | -const frame = session.waitFor("Page.screencastFrame", { timeoutMs: 10_000 }) // register first |
164 | | -await session.Page.startScreencast({ format: "png" }) |
165 | | -try { |
166 | | - const f = await frame // f.data is base64, same as captureScreenshot |
167 | | -} finally { |
168 | | - await session.Page.stopScreencast() // otherwise the cast stays open |
169 | | -} |
| 53 | +await session.Input.insertText({text: "hello"}) |
| 54 | +await session.Page.captureScreenshot({format: "png"}) |
170 | 55 | ``` |
171 | 56 |
|
172 | | -## Reusing code |
173 | | -The agent-workspace is per-project: `./.bcode/agent-workspace/`. |
174 | | -Use this to write memory files, scripts, and helper functions. |
175 | | -Imports work at any depth; pick whatever layout makes the project easiest to navigate. |
176 | | -
|
177 | | -```ts |
178 | | -// ./.bcode/agent-workspace/scrape_titles.ts (you write this with the `write` tool) |
179 | | -export async function scrapeTitles(session: any, urls: string[]) { |
180 | | - const titles: string[] = [] |
181 | | - await session.Page.enable() |
182 | | - for (const url of urls) { |
183 | | - const loaded = session.waitFor("Page.loadEventFired", { timeoutMs: 15_000 }) |
184 | | - await session.Page.navigate({ url }) |
185 | | - await loaded |
186 | | - const r = await session.Runtime.evaluate({ expression: "document.title", returnByValue: true }) |
187 | | - titles.push(r.result.value) |
188 | | - } |
189 | | - return titles |
190 | | -} |
191 | | -``` |
| 57 | +Register event waiters before the action that triggers them. Treat non-empty `Page.navigate.errorText` as failure. |
| 58 | +Every successful `Page.captureScreenshot` response is attached natively to the tool result; do not print or decode its |
| 59 | +base64. Persistent `ERR_TUNNEL_CONNECTION_FAILED` requires another source or browser, not retries on the same endpoint. |
| 60 | + |
| 61 | +## Reuse Code |
| 62 | + |
| 63 | +Write reusable modules under `./.bcode/agent-workspace/` and import them with a cache-busting query: |
192 | 64 |
|
193 | 65 | ```js |
194 | | -// later snippet |
195 | | -const path = process.cwd() + "/.bcode/agent-workspace/scrape_titles.ts" |
196 | | -// Cache-bust (`?t=${Date.now()}`) is your responsibility: without it, edits to the file won't be picked up. |
197 | | -const m = await import(`${path}?t=${Date.now()}`) |
198 | | -const titles = await m.scrapeTitles(session, ["https://example.com", "https://example.org"]) |
199 | | -console.log(JSON.stringify(titles)) |
| 66 | +const path = process.cwd() + "/.bcode/agent-workspace/helpers.ts" |
| 67 | +const helpers = await import(`${path}?t=${Date.now()}`) |
200 | 68 | ``` |
201 | 69 |
|
202 | | -## Guardrails |
203 | | -- Top-level `import` statements inside the snippet body are not allowed. Use `await import(...)` instead. |
204 | | -- No CPU-bound infinite loops without `await` — they ignore the timeout. Insert `await new Promise(r => setTimeout(r, 0))` to yield. |
205 | | -- `browser_execute` defaults to 60s; longer timeouts delay your next turn. After a timeout, only that call loses CDP access; the next call can continue on the same connection and target. The last command sent by the timed-out call may still finish. `Target.getTargets` succeeding means CDP is live; `session.connect()` is then a no-op, and reattaching the same target does not restart its renderer. |
206 | | -
|
207 | | -## Console |
208 | | -- `console.log`, `console.error`, `console.warn`, `console.info`, `console.debug` are all captured and streamed to the user. Treat them as your stdout. Other `console.*` methods write to bcode's stderr without being captured into the tool result. |
209 | | -- The snippet's `return` value is captured separately (JSON-serialized when possible). |
| 70 | +Use `await import(...)`; top-level static imports are unsupported. Avoid CPU-bound loops without await points. Prefer |
| 71 | +several small calls over one long call. After a timeout, continue in the next call if `Target.getTargets` still works. |
| 72 | +Before submitting, verify the current URL, selected entity or variant, counts, and every required field. |
0 commit comments