Skip to content

Commit 95c9f1c

Browse files
committed
feat(browser): specialize browser prompt
1 parent e634099 commit 95c9f1c

2 files changed

Lines changed: 53 additions & 280 deletions

File tree

  • packages
    • bcode-browser/skills/browser-execute
    • opencode/src/session/prompt
Lines changed: 40 additions & 177 deletions
Original file line numberDiff line numberDiff line change
@@ -1,209 +1,72 @@
11
---
22
name: browser-execute
3-
description: Use ONLY when calling the `browser_execute` tool or driving a real browser via the Chrome DevTools Protocol. Required reading before the first `browser_execute` call in a session. Covers the three connection methods (local Chrome with remote debugging, isolated debug-port profile, Browser Use cloud), the in-process `session` / `console` snippet model, attaching to a page target, common CDP commands, the per-project `.bcode/agent-workspace/` for reusable scripts, and screenshot auto-attachment.
3+
description: Required reference for driving a browser with browser_execute and its persistent CDP session.
44
---
55

6-
The `browser_execute` tool evaluates JavaScript against a connected browser `session` via the Chrome DevTools Protocol.
7-
The snippet runs in-process; `session` is bound to a long-lived CDP `Session` that persists.
8-
There is no helper namespace, just `session`, `console`, and standard JS globals.
6+
`browser_execute` runs JavaScript with `session`, `console`, and standard JS globals. The CDP `session` persists across
7+
calls, but local JavaScript variables do not. Use short deterministic snippets, print or return compact structured
8+
results, and checkpoint large collections under `./.bcode/agent-workspace/`.
99

10-
Workspace: `<projectRoot>/.bcode/agent-workspace/`. Read/write your reusable scripts here.
11-
Skills: `{{SKILLS_DIR}}/`. Read-only browser execute reference docs.
10+
This read-only skill is materialized under `{{SKILLS_DIR}}/browser-execute/` for the current run.
1211

13-
## Connecting
14-
In Browser Use Cloud API V4, `browser_execute` automatically connects and attaches the existing page once when the fresh run first uses this tool; do not call `session.connect()` or `session.use()` before driving it.
15-
Otherwise, call `session.connect(...)` once at the start of your work. There are three connection methods:
12+
## Connect
1613

17-
#### Way 1: connect to the user's running Chrome or Chromium-based browser (real profile, popup-gated).
18-
Choose when the task involves the user's logged-in sites, current browser state, cookies, saved data, etc.
14+
Browser Use Cloud API V4 automatically connects and attaches the existing page. When both `V4_RUN_ID` and
15+
`BU_CDP_WS` or `BU_CDP_URL` are set, start driving immediately; do not call `session.connect()` or `session.use()`.
1916

20-
```js
21-
// Attempts to connect to every detected Chrome, most-recently-launched first.
22-
await session.connect()
23-
```
24-
25-
For this to work the user must have navigated to `chrome://inspect/#remote-debugging` in their target Chrome and ticked "Allow remote debugging for this browser instance". This setting is per-profile and persists across every future launch of that profile. On Chrome 144 and later, the first attach also triggers an in-browser "Allow remote debugging?" popup that the user must click "Allow" on. The popup may reappear on later attaches under conditions that are not fully characterized — browser restart, time elapsed, new CDP session. Ask the user to click Allow again if a previously working connection starts 403'ing.
26-
27-
Failure modes:
28-
- `connect()` throws "No running browser with remote debugging detected". The checkbox at `chrome://inspect/#remote-debugging` has not been ticked in any running Chrome profile, or no Chrome is running.
29-
- `connect()` throws with "403" / "permission" / "WS closed before open". The checkbox is ticked but the user hasn't clicked Allow on the popup yet. By default `connect()` errors in 5s; pass `{ timeoutMs: 30000 }` to wait up to 30s for the click.
30-
31-
#### Way 2: connect to a Chrome or Chromium-based browser launched with a debug port (isolated profile, no popups).
32-
Choose for unattended automation, or for an isolated browser.
33-
34-
Launch Chrome with `--remote-debugging-port=<port> --user-data-dir=<path>`. Pick a directory you can access — e.g., a project-local one like `./.bcode/chrome-data-dir`.
35-
36-
```bash
37-
# Linux
38-
google-chrome --remote-debugging-port=9222 --user-data-dir=./.bcode/chrome-data-dir
39-
# macOS
40-
"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
41-
--remote-debugging-port=9222 --user-data-dir=./.bcode/chrome-data-dir
42-
# Windows (cmd.exe)
43-
"C:\Program Files\Google\Chrome\Application\chrome.exe" ^
44-
--remote-debugging-port=9222 --user-data-dir=.\.bcode\chrome-data-dir
45-
# Windows (PowerShell)
46-
& "C:\Program Files\Google\Chrome\Application\chrome.exe" `
47-
--remote-debugging-port=9222 --user-data-dir=.\.bcode\chrome-data-dir
48-
```
49-
50-
```js
51-
// Resolve the live WebSocket URL via `/json/version` and connect:
52-
const ver = await fetch("http://127.0.0.1:9222/json/version").then(r => r.json())
53-
await session.connect({ wsUrl: ver.webSocketDebuggerUrl })
54-
```
55-
56-
`--user-data-dir` must not be Chrome's platform default. Chrome 136 and later silently no-ops the `--remote-debugging-port` flag when `--user-data-dir` is the platform default. The platform defaults are `%LOCALAPPDATA%\Google\Chrome\User Data` on Windows, `~/Library/Application Support/Google/Chrome` on macOS, `~/.config/google-chrome` on Linux.
57-
You cannot reuse the user's everyday Chrome profile by copying its files into a custom directory.
58-
59-
Failure modes:
60-
- Chrome's launch log prints `DevTools listening on ws://...:<port>/...` immediately followed by `bind() failed: Address already in use` and Chrome exits. Confirm the port is actually open with `curl http://127.0.0.1:<port>/json/version` before connecting.
61-
- `{ profileDir }` raises ENOENT on `DevToolsActivePort`. Chrome 147+ doesn't write this file under custom `--user-data-dir`; use the `/json/version` route above instead.
62-
- Launch silently no-ops `--remote-debugging-port`. Launching a second Chrome that points at a `--user-data-dir` matching a running process ignores `--remote-debugging-port`.
63-
64-
#### Way 3: provision and connect to a Browser Use cloud browser.
65-
Choose when the computer has no browser or to work in the background.
66-
67-
```js
68-
// Provision and connect to a cloud browser
69-
const r = await fetch("https://api.browser-use.com/api/v3/browsers", {
70-
method: "POST",
71-
headers: { "X-Browser-Use-API-Key": process.env.BROWSER_USE_API_KEY, "Content-Type": "application/json" },
72-
body: "{}",
73-
})
74-
// Additional options: fetch https://docs.browser-use.com/cloud/api-v3/browsers/create-browser-session
75-
const { id, cdpUrl, liveUrl } = await r.json()
76-
const ver = await fetch(`${cdpUrl}/json/version`).then(r => r.json())
77-
await session.connect({ wsUrl: ver.webSocketDebuggerUrl })
78-
console.log("liveUrl for the user to watch:", liveUrl)
79-
```
80-
81-
```js
82-
// Browser Use cloud will eventually close idle browsers. An explicit stop frees the slot:
83-
await fetch(`https://api.browser-use.com/api/v3/browsers/${id}`, {
84-
method: "PATCH",
85-
headers: { "X-Browser-Use-API-Key": process.env.BROWSER_USE_API_KEY, "Content-Type": "application/json" },
86-
body: JSON.stringify({ action: "stop" }),
87-
})
88-
```
89-
90-
A cloud browser is a paid service and requires `BROWSER_USE_API_KEY` to be set in the environment.
91-
Browser Use cloud browsers auto-solve captchas. When you land on one, just stop driving and `await new Promise(r => setTimeout(r, 10000))`; the solver runs server-side and the page advances on its own.
92-
Browser Use has a free tier gated for intelligent and powerful agents. Unlimited free browsers and proxies, limited to 3 concurrent browsers. Agents can sign up and receive a key fully autonomously, but must pass a proof-of-agent-capability challenge. In can be done without any browser. To begin, post or webfetch https://browser-use.com and solve the challenge (see https://browser-use.com/llms.txt or POST https://api.browser-use.com/cloud/signup.).
93-
94-
#### Way 4: user-preconfigured endpoint
95-
Not a method you choose — a way for the user to hand you a pre-set endpoint.
96-
When `V4_RUN_ID` and `BU_CDP_WS` (or its alias `BU_CDP_URL`) are both set, `browser_execute` connects to that endpoint and attaches its existing non-internal page once before the first snippet. Go straight to driving it. Other environments keep the explicit connection flow, and explicit `{ wsUrl }` / `{ profileDir }` calls still connect to the requested endpoint instead.
97-
If that fixed endpoint closes or repeatedly fails its WebSocket upgrade, reconnecting to the same URL cannot recover it; the endpoint owner must replace it.
98-
99-
## Attaching to a target
100-
After connecting manually, attach to a page target before driving the browser. A preconfigured endpoint is already attached automatically:
17+
Otherwise connect once, then attach a non-internal page:
10118

10219
```js
20+
await session.connect()
10321
const targets = (await session.Target.getTargets({})).targetInfos
104-
// Pick the first non-internal tab if none was specified.
10522
const page = targets.find(t => t.type === "page" && !t.url.startsWith("chrome://"))
10623
await session.use(page.targetId)
10724
```
10825

109-
If a target-scoped command throws `CdpError` code `-32001` (`Session with given id not found`), the browser connection is still usable but the target session is stale. List targets again, `session.use(...)` the intended page, and retry the rejected command once. Calling `session.connect()` without arguments is a no-op while connected; it does not replace a stale target session.
26+
An explicit `{wsUrl}` connects to a chosen CDP endpoint. Every reconnect or browser switch clears the target attachment,
27+
so list targets and call `session.use(...)` again. If a target command throws `-32001`, reattach the intended page and
28+
retry once. Opening a tab does not switch the attachment.
11029

111-
Every explicit reconnect or browser switch retires the previous socket and clears its active target attachment. Re-list targets, call `session.use(...)`, and rediscover DOM nodes and Runtime objects before continuing.
30+
## Drive
11231

113-
Opening a tab creates a new `page` target but does not switch the active attachment. Call `Target.getTargets` again and `session.use(targetId)` when continuing there.
114-
115-
## Driving a page
116-
Domain methods follow `session.<Domain>.<method>(params)` and return Promises.
117-
The full surface (652 commands) is the Chrome DevTools Protocol.
118-
`Object.keys(session.domains).sort()` lists every CDP domain bound on the session; `Object.keys(session.Page).sort()` lists the methods for `Page`.
119-
For unknown param shapes, call with `{}` and inspect the thrown `CdpError` — `.data` carries the missing-field detail.
120-
121-
Common moves:
32+
CDP methods are available as `session.<Domain>.<method>(params)`. Inspect the surface with
33+
`Object.keys(session.domains).sort()` or `Object.keys(session.Page).sort()`.
12234

12335
```js
124-
// Navigate. Register the load waiter BEFORE navigate so a fast load isn't missed.
12536
await session.Page.enable()
126-
const loaded = session.waitFor("Page.loadEventFired", { timeoutMs: 15_000 })
127-
await session.Page.navigate({ url: "https://example.com" })
37+
const loaded = session.waitFor("Page.loadEventFired", {timeoutMs: 15_000})
38+
const navigation = await session.Page.navigate({url: "https://example.com"})
12839
await loaded
129-
// Page.navigate resolves even on network errors — its result carries `errorText` when the load failed.
40+
if (navigation.errorText) throw new Error(navigation.errorText)
13041

131-
// Evaluate JS in the page.
132-
const r = await session.Runtime.evaluate({
133-
expression: "document.title",
42+
const result = await session.Runtime.evaluate({
43+
expression: `JSON.stringify({title: document.title, text: document.body.innerText.slice(0, 4000)})`,
13444
returnByValue: true,
13545
})
136-
console.log(r.result.value)
137-
138-
// Click by coordinates.
139-
const x = 200, y = 300
140-
await session.Input.dispatchMouseEvent({ type: "mouseMoved", x, y })
141-
await session.Input.dispatchMouseEvent({ type: "mousePressed", x, y, button: "left", clickCount: 1 })
142-
await session.Input.dispatchMouseEvent({ type: "mouseReleased", x, y, button: "left", clickCount: 1 })
143-
144-
// Type text.
145-
await session.Input.insertText({ text: "hello" })
146-
147-
// Screenshot.
148-
await session.Page.captureScreenshot({ format: "png" })
149-
// You see the image inline on the next turn — `browser_execute` automatically
150-
// attaches every `Page.captureScreenshot` result. No need to decode, save, or
151-
// `read` the bytes back. The base64 is still in `data` (via the return value)
152-
// for the rare case you want to process it programmatically.
153-
```
154-
155-
`Page.navigate` can return a non-empty `errorText` instead of throwing. Treat it as a failed navigation. If `ERR_TUNNEL_CONNECTION_FAILED` persists, reloading, reattaching, or reconnecting to the same endpoint cannot change its proxy route; use another source or replace the cloud browser instead of retrying it.
46+
console.log(result.result.value)
15647

157-
Not auto-attaching a screenshot: attachment is keyed on the `Page.captureScreenshot` response; pixels that arrive as an event are never attached.
48+
await session.Runtime.evaluate({
49+
expression: `document.querySelector("button")?.click()`,
50+
returnByValue: true,
51+
})
15852

159-
```js
160-
await session.Page.enable()
161-
// Optional: screencast has no `clip`, so override the viewport for exact frame size.
162-
await session.Emulation.setDeviceMetricsOverride({ width: 1200, height: 630, deviceScaleFactor: 1, mobile: false })
163-
const frame = session.waitFor("Page.screencastFrame", { timeoutMs: 10_000 }) // register first
164-
await session.Page.startScreencast({ format: "png" })
165-
try {
166-
const f = await frame // f.data is base64, same as captureScreenshot
167-
} finally {
168-
await session.Page.stopScreencast() // otherwise the cast stays open
169-
}
53+
await session.Input.insertText({text: "hello"})
54+
await session.Page.captureScreenshot({format: "png"})
17055
```
17156

172-
## Reusing code
173-
The agent-workspace is per-project: `./.bcode/agent-workspace/`.
174-
Use this to write memory files, scripts, and helper functions.
175-
Imports work at any depth; pick whatever layout makes the project easiest to navigate.
176-
177-
```ts
178-
// ./.bcode/agent-workspace/scrape_titles.ts (you write this with the `write` tool)
179-
export async function scrapeTitles(session: any, urls: string[]) {
180-
const titles: string[] = []
181-
await session.Page.enable()
182-
for (const url of urls) {
183-
const loaded = session.waitFor("Page.loadEventFired", { timeoutMs: 15_000 })
184-
await session.Page.navigate({ url })
185-
await loaded
186-
const r = await session.Runtime.evaluate({ expression: "document.title", returnByValue: true })
187-
titles.push(r.result.value)
188-
}
189-
return titles
190-
}
191-
```
57+
Register event waiters before the action that triggers them. Treat non-empty `Page.navigate.errorText` as failure.
58+
Every successful `Page.captureScreenshot` response is attached natively to the tool result; do not print or decode its
59+
base64. Persistent `ERR_TUNNEL_CONNECTION_FAILED` requires another source or browser, not retries on the same endpoint.
60+
61+
## Reuse Code
62+
63+
Write reusable modules under `./.bcode/agent-workspace/` and import them with a cache-busting query:
19264

19365
```js
194-
// later snippet
195-
const path = process.cwd() + "/.bcode/agent-workspace/scrape_titles.ts"
196-
// Cache-bust (`?t=${Date.now()}`) is your responsibility: without it, edits to the file won't be picked up.
197-
const m = await import(`${path}?t=${Date.now()}`)
198-
const titles = await m.scrapeTitles(session, ["https://example.com", "https://example.org"])
199-
console.log(JSON.stringify(titles))
66+
const path = process.cwd() + "/.bcode/agent-workspace/helpers.ts"
67+
const helpers = await import(`${path}?t=${Date.now()}`)
20068
```
20169

202-
## Guardrails
203-
- Top-level `import` statements inside the snippet body are not allowed. Use `await import(...)` instead.
204-
- No CPU-bound infinite loops without `await` — they ignore the timeout. Insert `await new Promise(r => setTimeout(r, 0))` to yield.
205-
- `browser_execute` defaults to 60s; longer timeouts delay your next turn. After a timeout, only that call loses CDP access; the next call can continue on the same connection and target. The last command sent by the timed-out call may still finish. `Target.getTargets` succeeding means CDP is live; `session.connect()` is then a no-op, and reattaching the same target does not restart its renderer.
206-
207-
## Console
208-
- `console.log`, `console.error`, `console.warn`, `console.info`, `console.debug` are all captured and streamed to the user. Treat them as your stdout. Other `console.*` methods write to bcode's stderr without being captured into the tool result.
209-
- The snippet's `return` value is captured separately (JSON-serialized when possible).
70+
Use `await import(...)`; top-level static imports are unsupported. Avoid CPU-bound loops without await points. Prefer
71+
several small calls over one long call. After a timeout, continue in the next call if `Target.getTargets` still works.
72+
Before submitting, verify the current URL, selected entity or variant, counts, and every required field.

0 commit comments

Comments
 (0)