Add in-app browser host and surface management

.tmp-agent-handoff.md

@@ -0,0 +1,174 @@
+# Agent Handoff: Ryco In-App Browser Computer-Use
+
+## Repo / branch
+
+- **Path:** `/Users/laurinfrank/.ryco/worktrees/0ed0fd07-3a8e-4874-8a7d-f15a17a80f5a/feature-add-in-app-browser-usability__budop`
+- **Branch:** `feature/add-in-app-browser-usability`
+- **Status:** ~56 modified/new files, **nothing committed yet**
+
+## Goal
+
+Finish the in-app browser so agents can do real **computer use** (open browser → observe page → click/type → verify) through Ryco's isolated Electron browser, across providers (Codex, Claude, Copilot, OpenCode, Cursor).
+
+Design spec: `docs/superpowers/specs/2026-06-22-in-app-browser-design.md`
+
+**AGENTS.md requirements before done:** `bun fmt`, `bun lint`, `bun typecheck`, `bun run test` (not `bun test`).
+
+---
+
+## What's already implemented
+
+### Desktop host (`apps/desktop/src/browser/`)
+
+- `BrowserKernel.ts`: navigation, input (x/y + **DOM ref click**), screenshot (`capturePage`), structured DOM snapshot (refs + bounds), console/network buffers
+- `BrowserObservationHelpers.ts`: DOM snapshot script, console parsing
+- `BrowserHostConnection.ts`: `screenshots: true` capability
+- `BrowserSurfaceManager.ts`: HiDPI bounds via `resolveElectronSurfaceBounds` from `@ryco/shared/browser`
+
+### Server control plane (`apps/server/src/browser/`)
+
+- `BrowserService.ts`: observation methods (`snapshotDom`, `screenshot`, `readConsole`, `readNetwork`, `waitFor`), agent session cleanup (`closeAgentSessionsForThread`), origin policy enforcement (`source: "agent"` blocks `"ask"` origins)
+- `BrowserArtifactStore.ts`: PNG/JSON blob storage under `{stateDir}/browser-artifacts/` with `readData`
+- `BrowserMcpBridge.ts` + `browser-mcp-stdio.ts` + `BrowserRuntimeMcpConfig.ts`: MCP bridge for OpenCode/Cursor subprocesses
+- `BrowserThreadCleanup.ts`: stops MCP bridge + closes agent sessions after provider turn
+- `server.ts`: wires `BrowserLayerLive` (mergeAll: HostRegistry, Policy, ArtifactStore, McpBridge, BrowserService, ProviderRuntimeToolRegistry) + `ProviderRuntimeEventHubLive` in ProviderLayer
+
+### Provider tool bridge (`apps/server/src/provider/tools/`)
+
+- `BrowserRuntimeTool.ts`: **12 tools** — `browser_open`, `browser_navigate`, back/forward/reload/stop, `browser_input`, `browser_snapshot`, `browser_screenshot`, `browser_console`, `browser_network`, `browser_wait_for`
+- `ProviderRuntimeToolRegistry.ts`: executes tools, emits `browser_tool_call` lifecycle events via `ProviderRuntimeEventHub`
+- `BrowserRuntimeToolHelpers.ts`: result mapping + **PNG image content blocks** for screenshots (`enrichBrowserRuntimeToolCallToolResult`)
+- `BrowserToolLifecycleEvents.ts`: canonical `browser_tool_call` started/completed events
+
+### Provider integrations
+
+| Provider | Status | How |
+|----------|--------|-----|
+| **Claude** | ✅ wired | `ClaudeBrowserRuntimeTools.ts` — in-process MCP server `ryco`, auto-allow in `canUseTool` |
+| **Copilot** | ✅ wired | `CopilotBrowserRuntimeTools.ts` — `defineTool`, negative system prompt removed |
+| **OpenCode** | ✅ wired | MCP bridge at session start |
+| **Cursor/ACP** | ✅ wired | Ryco browser MCP injected into `mcpServers` via `AcpSessionRuntime` |
+| **Codex** | ❌ blocked | `item/tool/call` handler in `CodexSessionRuntime.ts` exists, but `resolveProviderBrowserToolSupport("codex")` returns **`supported: false`** — Codex app-server has no `dynamicTools` field on thread start yet |
+
+### Web UI (`apps/web/src/components/`)
+
+- `BrowserPanel.tsx`: thread profile mode, agent session badge, permission/download toasts, HiDPI surface bounds
+- `BrowserPanel.logic.ts`: session adoption helpers, tests extended
+
+### Contracts (`packages/contracts/src/browser.ts`, `rpc.ts`)
+
+- `BrowserInputAction` supports `{ type: "click", ref }` in addition to x/y
+- Observation result schemas, WS RPC methods for snapshot/screenshot/console/network/waitFor
+
+### Tests added/updated (passing in targeted runs)
+
+- `BrowserService.test.ts`, `BrowserArtifactStore.test.ts`, `BrowserRuntimeTool.test.ts`, `BrowserRuntimeToolHelpers.test.ts`, `ClaudeBrowserRuntimeTools.test.ts`, `BrowserPanel.logic.test.ts`
+- Adapter test layer mocks via `BrowserRuntimeToolTestLayers.ts`
+
+---
+
+## What's NOT done / known gaps
+
+1. **`bun typecheck` fails** — production code is clean (0 non-test errors), but ~168 errors in `apps/server/src/server.test.ts` and `apps/server/src/cli.test.ts` (Effect `unknown` in requirements channel from new browser layers). **AGENTS.md requires full typecheck pass.**
+
+2. **Full test suite not re-verified** after last fixes — targeted browser tests pass; full `bun run test` in `apps/server` was interrupted. A bad `Layer.mergeAll` in `server.test.ts` briefly broke 100 tests; fixed back to `BrowserServiceLive.pipe(Layer.provideMerge(...), Layer.provideMerge(browserRuntimeToolTestLayers))`.
+
+3. **Codex non-functional** — inbound `item/tool/call` handler is dead code until Codex schema supports advertising tools at thread start. Either regenerate `effect-codex-app-server` when upstream adds `dynamicTools`, or keep `supported: false` with a clear reason.
+
+4. **Origin approval UX missing** — agent navigation to `"ask"` origins hard-fails with `origin_denied`; no `browser_origin_approval` request.opened/resolved flow in UI.
+
+5. **Single-tab / no popups** — `BrowserKernel` only drives `tabs[0]`; `window.open` denied. OAuth/popup flows break.
+
+6. **Session lifecycle** — cleanup on turn end exists, but panel unmount doesn't call `closeSession`; no TTL for abandoned WebContentsViews.
+
+7. **`runServer` uses `as` cast** in `server.ts` (line ~495) instead of `satisfies` — layer graph leaks `unknown` at type level; runtime works.
+
+8. **Timeline UI** — `browser_tool_call` events emitted server-side; orchestration/timeline rendering may still be incomplete.
+
+9. **Nothing committed** — user has not asked for a commit yet.
+
+---
+
+## Your task: finish to merge-ready
+
+Follow `AGENTS.md`: `bun fmt`, `bun lint`, `bun typecheck`, `bun run test` must all pass before done.
+
+### Priority 0 — Must do before merge
+
+- [ ] Fix ~168 test typecheck errors in `server.test.ts` / `cli.test.ts`:
+  - Likely fix: align `buildAppUnderTest` browser layer with production pattern OR use `browserRuntimeToolTestLayers` mocks consistently
+  - Do **not** use `Layer.mergeAll` for browser services in tests without proper `Layer.provide` — caused `Service not found: BrowserHostRegistry`
+- [ ] Run `bun fmt && bun lint && bun typecheck && bun run test` — all must pass
+- [ ] Verify no accidental file loss from git stash/checkout during prior session (confirm `BrowserService.ts` ~994 lines, `BrowserArtifactStore.ts` has `readData` + fs blobs)
+
+### Priority 1 — Strongly recommended
+
+- [ ] Re-run full server test suite (1440+ tests) and fix failures
+- [ ] Manual smoke: desktop app → open Browser panel → agent on Claude drives `browser_open` → `browser_snapshot` → ref click → `browser_screenshot`
+- [ ] Clean up `runServer` typing if possible without breaking launch layer contract in `server.ts:493`
+
+### Priority 2 — Nice to have / follow-up
+
+- [ ] Codex: investigate `effect-codex-app-server` for dynamic tool registration; flip `resolveProviderBrowserToolSupport("codex")` when possible
+- [ ] Origin approval UI for agent `"ask"` navigations (`browser_origin_approval` request type exists in contracts)
+- [ ] Multi-tab / controlled popup support for OAuth
+- [ ] Timeline UI for `browser_tool_call` items in chat
+
+---
+
+## Key files to read first
+
+```
+apps/server/src/server.ts                          # layer wiring
+apps/server/src/browser/BrowserService.ts          # observation + policy
+apps/server/src/provider/tools/BrowserRuntimeTool.ts
+apps/server/src/provider/tools/ProviderRuntimeToolRegistry.ts
+apps/server/src/provider/tools/ClaudeBrowserRuntimeTools.ts
+apps/server/src/provider/tools/CopilotBrowserRuntimeTools.ts
+apps/server/src/provider/Layers/CodexSessionRuntime.ts
+apps/server/src/browser/BrowserMcpBridge.ts
+apps/desktop/src/browser/BrowserKernel.ts
+apps/web/src/components/BrowserPanel.tsx
+apps/server/src/server.test.ts                     # fix test layer + typecheck (~line 610 browserLayer, ~line 419 buildAppUnderTest)
+docs/superpowers/specs/2026-06-22-in-app-browser-design.md
+docs/superpowers/plans/2026-06-22-in-app-browser.md
+```
+
+---
+
+## Architecture (quick ref)
+
+```
+Agent tool call
+  → Provider adapter (Claude MCP / Copilot defineTool / OpenCode+Cursor MCP bridge / Codex item/tool/call)
+  → ProviderRuntimeToolRegistry.executeBrowserTool
+  → BrowserRuntimeTool → BrowserService
+  → BrowserHostRegistry → Desktop BrowserKernel (WebContentsView)
+  → observation back (snapshot/screenshot/console) via BrowserArtifactStore
+  → browser_tool_call lifecycle events → ProviderRuntimeEventHub
+```
+
+Two channels:
+
+- **UI**: `browser.*` WS RPC + `desktopBridge.browser` surface IPC
+- **Agents**: provider-specific MCP/tools → shared registry → same `BrowserService`
+
+---
+
+## Pitfalls from prior session
+
+- **Don't `git stash push -u` carelessly** — lost untracked files twice
+- **Don't use `Layer.mergeAll` for browser layer in tests** without dependency ordering — use `BrowserServiceLive.pipe(Layer.provideMerge(...))` pattern
+- **`BrowserServiceLive` hard `yield* BrowserArtifactStore` at layer init** caused `unknown` leak — use `Effect.serviceOption` at call time in methods, not layer init (partially done)
+- **`server.test.ts` browser layer** should use `browserRuntimeToolTestLayers` mock, not live `BrowserMcpBridgeLive`
+- **`runServer satisfies`** fails due to layer `unknown` — production uses `as` cast as workaround
+
+---
+
+## Success criteria
+
+- [ ] `bun fmt` / `bun lint` / `bun typecheck` / `bun run test` all green
+- [ ] Claude (and ideally Copilot/Cursor/OpenCode) can drive browser tools on desktop
+- [ ] Codex either works or is explicitly documented/typed as unsupported with clear reason
+- [ ] No regressions in `server.test.ts` integration tests
+- [ ] User decides when to commit (don't commit unless asked)

apps/desktop/src/browser/BrowserHost.ts

@@ -0,0 +1,22 @@
+import { BrowserHostConnection } from "./BrowserHostConnection.ts";
+import { BrowserKernel } from "./BrowserKernel.ts";
+
+export class BrowserHost {
+  readonly kernel = new BrowserKernel();
+  readonly connection: BrowserHostConnection;
+
+  constructor(appRunId: string) {
+    this.connection = new BrowserHostConnection(appRunId, (command) =>
+      this.kernel.execute(command),
+    );
+    this.kernel.setEventSink((event) => this.connection.publishEvent(event));
+  }
+
+  start(input: { readonly wsBaseUrl: string; readonly token: string }): Promise<void> {
+    return this.connection.start(input);
+  }
+
+  stop(): Promise<void> {
+    return this.connection.stop();
+  }
+}

apps/desktop/src/browser/BrowserHostConnection.ts

@@ -0,0 +1,178 @@
+import * as NodeSocket from "@effect/platform-node/NodeSocket";
+import {
+  BrowserHostId,
+  BrowserHostRpcGroup,
+  BrowserHostRunId,
+  BROWSER_HOST_METHODS,
+} from "@ryco/contracts";
+import type {
+  BrowserEvent,
+  BrowserHostCommandEnvelope,
+  BrowserCommandResult,
+} from "@ryco/contracts";
+import { Effect, Exit, Layer, ManagedRuntime, Scope, Stream } from "effect";
+import { RpcClient, RpcSerialization } from "effect/unstable/rpc";
+import * as Socket from "effect/unstable/socket/Socket";
+
+const makeBrowserHostRpcClient = RpcClient.make(BrowserHostRpcGroup);
+type BrowserHostRpcClient =
+  typeof makeBrowserHostRpcClient extends Effect.Effect<infer Client, any, any> ? Client : never;
+
+type CommandHandler = (command: BrowserHostCommandEnvelope) => Promise<BrowserCommandResult>;
+
+function browserHostWsUrl(wsBaseUrl: string): string {
+  const url = new URL(wsBaseUrl);
+  url.pathname = "/browser-host/ws";
+  url.search = "";
+  url.hash = "";
+  return url.toString();
+}
+
+function browserHostProtocolLayer(
+  wsUrl: string,
+  token: string,
+): Layer.Layer<RpcClient.Protocol, never, never> {
+  const webSocketConstructorLayer = Layer.succeed(
+    Socket.WebSocketConstructor,
+    (socketUrl, protocols) =>
+      new NodeSocket.NodeWS.WebSocket(socketUrl, protocols, {
+        headers: {
+          authorization: `Bearer ${token}`,
+        },
+      }) as unknown as globalThis.WebSocket,
+  );
+
+  return RpcClient.layerProtocolSocket().pipe(
+    Layer.provide(Socket.layerWebSocket(wsUrl).pipe(Layer.provide(webSocketConstructorLayer))),
+    Layer.provide(RpcSerialization.layerJson),
+  );
+}
+
+export class BrowserHostConnection {
+  private runtime: ManagedRuntime.ManagedRuntime<RpcClient.Protocol, never> | null = null;
+  private client: BrowserHostRpcClient | null = null;
+  private clientScope: Scope.Scope | null = null;
+  private heartbeatTimer: ReturnType<typeof setInterval> | null = null;
+  private commandStreamAbort = false;
+  private readonly handleCommand: CommandHandler;
+
+  readonly hostId: BrowserHostId;
+  readonly runId: BrowserHostRunId;
+
+  constructor(appRunId: string, handleCommand: CommandHandler) {
+    this.handleCommand = handleCommand;
+    this.hostId = BrowserHostId.make(`desktop-browser-host:${appRunId}`);
+    this.runId = BrowserHostRunId.make(`desktop-browser-host-run:${crypto.randomUUID()}`);
+  }
+
+  async start(input: { readonly wsBaseUrl: string; readonly token: string }): Promise<void> {
+    await this.stop();
+    this.commandStreamAbort = false;
+    const runtime = ManagedRuntime.make(
+      browserHostProtocolLayer(browserHostWsUrl(input.wsBaseUrl), input.token),
+    );
+    this.runtime = runtime;
+    const clientScope = runtime.runSync(Scope.make());
+    this.clientScope = clientScope;
+    const client = await runtime.runPromise(Scope.provide(clientScope)(makeBrowserHostRpcClient));
+    this.client = client;
+    await runtime.runPromise(
+      client[BROWSER_HOST_METHODS.register]({
+        hostId: this.hostId,
+        runId: this.runId,
+        capabilities: {
+          surface: true,
+          persistentProfiles: true,
+          temporaryProfiles: true,
+          screenshots: true,
+          domSnapshot: true,
+          input: true,
+          downloads: false,
+          devtools: false,
+        },
+      }),
+    );
+    this.startCommandStream(client);
+    this.heartbeatTimer = setInterval(() => {
+      void this.heartbeat().catch((error) => {
+        console.warn("[desktop-browser-host] heartbeat failed", error);
+      });
+    }, 5_000);
+    this.heartbeatTimer.unref();
+  }
+
+  async stop(): Promise<void> {
+    this.commandStreamAbort = true;
+    if (this.heartbeatTimer) {
+      clearInterval(this.heartbeatTimer);
+      this.heartbeatTimer = null;
+    }
+    const runtime = this.runtime;
+    const clientScope = this.clientScope;
+    this.runtime = null;
+    this.client = null;
+    this.clientScope = null;
+    if (runtime) {
+      if (clientScope) {
+        await runtime.runPromise(Scope.close(clientScope, Exit.void)).catch(() => undefined);
+      }
+      await runtime.dispose();
+    }
+  }
+
+  async publishEvent(event: BrowserEvent): Promise<void> {
+    const runtime = this.runtime;
+    const client = this.client;
+    if (!runtime || !client) return;
+    await runtime.runPromise(
+      client[BROWSER_HOST_METHODS.event]({
+        hostId: this.hostId,
+        runId: this.runId,
+        event,
+      }),
+    );
+  }
+
+  private async heartbeat(): Promise<void> {
+    const runtime = this.runtime;
+    const client = this.client;
+    if (!runtime || !client) return;
+    await runtime.runPromise(
+      client[BROWSER_HOST_METHODS.heartbeat]({
+        hostId: this.hostId,
+        runId: this.runId,
+      }),
+    );
+  }
+
+  private startCommandStream(client: BrowserHostRpcClient): void {
+    const runtime = this.runtime;
+    if (!runtime) return;
+    void runtime
+      .runPromise(
+        Stream.runForEach(
+          client[BROWSER_HOST_METHODS.subscribeCommands]({
+            hostId: this.hostId,
+            runId: this.runId,
+          }),
+          (envelope) =>
+            Effect.promise(async () => {
+              if (this.commandStreamAbort) return;
+              const result = await this.handleCommand(envelope);
+              await runtime.runPromise(
+                client[BROWSER_HOST_METHODS.commandResult]({
+                  hostId: this.hostId,
+                  runId: this.runId,
+                  result,
+                }),
+              );
+            }),
+        ),
+      )
+      .catch((error) => {
+        if (!this.commandStreamAbort) {
+          console.warn("[desktop-browser-host] command stream closed", error);
+        }
+      });
+  }
+}