Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
{
"$schema": "https://anthropic.com/claude-code/marketplace.schema.json",
"name": "open-computer-use",
"description": "macOS computer use plugin — MCP server and agent skill for controlling real logged-in apps",
"description": "Windows and macOS computer use plugin — MCP server and agent skill for controlling real logged-in apps",
"owner": {
"name": "nogu66"
},
"plugins": [
{
"name": "open-computer-use",
"source": "./",
"description": "Control logged-in Chrome, Slack, and native macOS apps from Claude Code, Codex, or Cursor",
"description": "Control logged-in desktop apps from Claude Code, Codex, or Cursor",
"version": "0.1.0",
"author": {
"name": "nogu66"
Expand Down
2 changes: 1 addition & 1 deletion .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "open-computer-use",
"description": "macOS computer use for AI agents — control logged-in Chrome and native apps via Accessibility API and MCP",
"description": "Windows and macOS computer use for AI agents via Microsoft UI Automation or macOS Accessibility API",
"version": "0.1.0",
"author": {
"name": "nogu66"
Expand Down
8 changes: 4 additions & 4 deletions .codex-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -1,19 +1,19 @@
{
"name": "open-computer-use",
"version": "0.1.0",
"description": "macOS computer use for AI agents — control logged-in Chrome and native apps via Accessibility API and MCP",
"description": "Windows and macOS computer use for AI agents via Microsoft UI Automation or macOS Accessibility API",
"author": {
"name": "nogu66"
},
"repository": "https://github.com/nogu66/open-computer-use",
"license": "MIT",
"keywords": ["macos", "computer-use", "accessibility", "mcp", "browser-automation"],
"keywords": ["windows", "macos", "computer-use", "accessibility", "msuia", "mcp", "browser-automation"],
"skills": "./skills/",
"mcpServers": "./.mcp.json",
"interface": {
"displayName": "open-computer-use",
"shortDescription": "Control your logged-in Mac apps from Codex",
"longDescription": "Operate the apps you already have open — logged-in Chrome, Slack, native macOS apps — via Accessibility API and synthetic input. Exposed as MCP tools and a Bash CLI.",
"shortDescription": "Control logged-in desktop apps from Codex",
"longDescription": "Operate the apps you already have open. Windows uses Microsoft UI Automation through PowerShell; macOS keeps the Accessibility API and synthetic input path. Destructive typed commands and delete-like UI actions are blocked by default on Windows.",
"developerName": "nogu66",
"category": "Productivity"
}
Expand Down
10 changes: 8 additions & 2 deletions .cursor/mcp.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
{
"mcpServers": {
"open-computer-use": {
"command": "${workspaceFolder}/scripts/mcp-server.sh",
"args": []
"command": "powershell.exe",
"args": [
"-NoProfile",
"-ExecutionPolicy",
"Bypass",
"-File",
"${workspaceFolder}/scripts/mcp-server.ps1"
]
}
}
}
8 changes: 8 additions & 0 deletions .cursor/mcp.macos.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"mcpServers": {
"open-computer-use": {
"command": "${workspaceFolder}/scripts/mcp-server.sh",
"args": []
}
}
}
101 changes: 24 additions & 77 deletions .cursor/skills/open-computer-use/SKILL.md
Original file line number Diff line number Diff line change
@@ -1,99 +1,46 @@
---
name: open-computer-use
description: Use whenever the user wants to control a macOS GUI app — clicking buttons, reading on-screen content, typing into open windows, or operating their logged-in browser. Triggers on "Chrome を操作", "ログイン済みのブラウザで", "Mac アプリを動かして", "ocu", "computer use", "operate the browser". Prefer over Playwright when login state, cookies, SSO, passkeys, or extensions matter.
description: Use whenever the user wants to control a real desktop GUI app through MCP or CLI. On Windows use Microsoft UI Automation via scripts/ocu-windows.ps1; on macOS use the Swift Accessibility API binary. Prefer when login state, cookies, SSO, passkeys, extensions, or native app state matter.
---

# open-computer-use (`ocu`)

macOS Accessibility API + CGEvent + `screencapture` packaged as one CLI/MCP binary. Model-agnostic: works from any agent that can call Bash or MCP.
Desktop computer use through native accessibility APIs.

## Install (latest release)
- Windows: PowerShell + Microsoft UI Automation (`scripts/ocu-windows.ps1`)
- macOS: Swift + Accessibility API + CGEvent + `screencapture`

```bash
curl -fsSL https://raw.githubusercontent.com/nogu66/open-computer-use/main/scripts/install.sh | bash
export PATH="$HOME/.local/bin:$PATH"
ocu --version
```

Pinned version: `OCU_VERSION=v0.1.0 ./scripts/install.sh`
Source build only: `./scripts/install.sh --from-source`

## Binary resolution

When installed as a plugin, prefer the bundled wrapper:
## Windows

```bash
OCU="${CLAUDE_PLUGIN_ROOT}/scripts/ocu-cli.sh"
# Codex also sets PLUGIN_ROOT; Cursor project checkout:
# OCU="$(git rev-parse --show-toplevel)/scripts/ocu-cli.sh"
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\smoke-test-windows.ps1
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\ocu-windows.ps1 apps
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\ocu-windows.ps1 tree --app-id chrome --depth 8
```

After install:
Windows MCP is launched with:

```bash
OCU="$(command -v ocu)"
```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\mcp-server.ps1
```

Wrappers auto-install the latest release to `~/.local/bin/ocu` on first use unless `OCU_SKIP_AUTO_INSTALL=1`.

## When to use

- Logged-in Chrome, Gmail, Notion, Slack, banking sites
- Native Mac apps without APIs (Notes, System Settings, etc.)
- Reading what is actually on screen
- Avoiding bot detection (no CDP, no new browser profile)

## When not to use

- Fresh-profile scraping only → Playwright
- Public HTML fetch only → WebFetch / curl
- Service has a stable API → use the API
Use `app_id`, `process_name`, `pid`, or `title`. `bundle_id` is still accepted as
an alias for clients that reuse the macOS schema.

## Core workflow

```bash
$OCU apps
$OCU activate --bundle-id com.google.Chrome
$OCU tree --bundle-id com.google.Chrome --depth 8
$OCU click --bundle-id com.google.Chrome --query "Address and search bar"
$OCU type --text "https://example.com"
$OCU key --key return
$OCU shot --bundle-id com.google.Chrome --out /tmp/after.png
```

## Subcommands

| Command | Purpose |
|---|---|
| `apps` | List GUI apps (bundle ID + PID) |
| `activate --bundle-id <id>` | Bring app to front (**call first**) |
| `tree --bundle-id <id> [--depth N]` | AX tree; add `--json` for structured output |
| `find --bundle-id <id> --query <q>` | Substring match on AX labels |
| `wait --bundle-id <id> --query <q> [--timeout SEC]` | Poll until element appears |
| `click` / `rclick` | Left / right click by `--query` |
| `type --text <s>` | Type into focused field |
| `key --key <name> [--mods cmd,shift,alt,ctrl]` | Key press |
| `scroll` | Scroll at element or cursor |
| `menu --bundle-id <id> --path <p>` | Menubar path, e.g. `"File/New Tab"` |
| `shot [--bundle-id <id>] [--out <path>]` | Screenshot PNG |
| `clip get` / `clip set --text <s>` | Clipboard |

## MCP vs CLI
1. `list_apps` / `apps` to identify the target.
2. `activate` before typing.
3. `get_ax_tree` / `tree` or `ax_tree_json` to inspect visible UI.
4. Use `click_element`, `click_ref`, `type_text`, `key_press`, `wait_for`, and `screenshot`.
5. Verify visually when labels are ambiguous.

Same binary. Plugin install exposes MCP tools (`list_apps`, `get_ax_tree`, `click_element`, …). Bash agents use the CLI subcommands above.

Direct MCP (no wrapper):

```bash
claude mcp add open-computer-use -- $(command -v ocu)
```
## Windows safety policy

## Pitfalls
The Windows path blocks destructive typed commands, unsafe paste, Enter after an
unsafe key-by-key command, and delete-like UI labels by default.

- Always `activate` first — otherwise `type`/`key` leak to the terminal
- Prefer `menu` over `Cmd+T` when Chrome is playing video (keys may be captured)
- `find`/`click` return the first match — narrow queries or inspect `tree` first
- CLI mode has no `@eN` refs — use `--query` (MCP `click_ref` uses refs from last tree)
- Grant **Accessibility** (and **Screen Recording** for screenshots) to the parent app (Claude Code, Cursor, Codex, Terminal)
Unsafe override requires both `OCU_ALLOW_UNSAFE_INPUT=1` and
`allow_unsafe=true`.

Repository: https://github.com/nogu66/open-computer-use
1 change: 1 addition & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@

- [ ] `swift build` passes
- [ ] `swift test` passes
- [ ] Windows smoke passes (`scripts\smoke-test-windows.ps1`) if Windows-facing
- [ ] Manually exercised on macOS (describe steps)

## Screenshots / transcripts (optional)
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -66,3 +66,23 @@ jobs:
else
echo "swift-format not available; skipping"
fi

windows-smoke:
name: Windows PowerShell smoke
runs-on: windows-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Validate JSON configs
shell: pwsh
run: |
Get-ChildItem -Recurse -Filter *.json |
ForEach-Object {
Get-Content -Raw -LiteralPath $_.FullName | ConvertFrom-Json | Out-Null
Write-Host "OK $($_.FullName)"
}

- name: Run Windows MCP smoke test
shell: powershell
run: powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\smoke-test-windows.ps1
10 changes: 8 additions & 2 deletions .mcp.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,14 @@
{
"mcpServers": {
"open-computer-use": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/mcp-server.sh",
"args": []
"command": "powershell.exe",
"args": [
"-NoProfile",
"-ExecutionPolicy",
"Bypass",
"-File",
"${CLAUDE_PLUGIN_ROOT}/scripts/mcp-server.ps1"
]
}
}
}
8 changes: 8 additions & 0 deletions .mcp.macos.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{
"mcpServers": {
"open-computer-use": {
"command": "${CLAUDE_PLUGIN_ROOT}/scripts/mcp-server.sh",
"args": []
}
}
}
10 changes: 9 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,15 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Added

- Nothing yet.
- Windows MCP/CLI path using PowerShell and Microsoft UI Automation.
- Guardrails for destructive typed commands, unsafe paste, file modification
commands, recursive deletion, and delete-like UI labels on Windows.
- Windows MCP smoke test script and GitHub Actions Windows smoke job.

### Changed

- `Package.swift` now exposes the macOS `ocu` executable target only on macOS so
`OCUCore` can build and test on Windows.

## [0.1.0] - 2026-05-22

Expand Down
50 changes: 30 additions & 20 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,39 @@
//
import PackageDescription

var products: [Product] = [
.library(name: "OCUCore", targets: ["OCUCore"])
]

var targets: [Target] = [
.target(
name: "OCUCore",
path: "Sources/OCUCore"
),
.testTarget(
name: "OCUCoreTests",
dependencies: ["OCUCore"],
path: "Tests/OCUCoreTests"
)
]

#if os(macOS)
products.insert(.executable(name: "ocu", targets: ["ocu"]), at: 0)
targets.insert(
.executableTarget(
name: "ocu",
dependencies: ["OCUCore"],
path: "Sources/ocu"
),
at: 1
)
#endif

let package = Package(
name: "OpenComputerUse",
platforms: [
.macOS(.v13)
],
products: [
.executable(name: "ocu", targets: ["ocu"]),
.library(name: "OCUCore", targets: ["OCUCore"])
],
targets: [
.target(
name: "OCUCore",
path: "Sources/OCUCore"
),
.executableTarget(
name: "ocu",
dependencies: ["OCUCore"],
path: "Sources/ocu"
),
.testTarget(
name: "OCUCoreTests",
dependencies: ["OCUCore"],
path: "Tests/OCUCoreTests"
)
]
products: products,
targets: targets
)
33 changes: 28 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@

<a href="README.ja.md">日本語</a>

**macOS computer use for AI agents** — control the apps you already have open (logged-in Chrome, Slack, native apps) via the Accessibility API and synthetic input, exposed as an MCP stdio server and a small CLI.
**Desktop computer use for AI agents** - control the apps you already have open (logged-in Chrome, Slack, native apps) via native accessibility APIs, exposed as an MCP stdio server and a small CLI.

This repository keeps the original macOS Accessibility API path and adds a Windows
path based on Microsoft UI Automation. Windows input is guarded by default:
dangerous shell text, recursive deletion, file modification commands, encoded
PowerShell, and delete-like UI labels are blocked unless the operator explicitly
enables the two-step unsafe override.

Inspired by [Codex Computer Use](https://developers.openai.com/codex/app/computer-use): same idea (OS-level AX + CGEvent, not CDP), packaged for Claude Code, Cursor, Codex, and any MCP client.

Expand All @@ -11,7 +17,24 @@ Inspired by [Codex Computer Use](https://developers.openai.com/codex/app/compute
| Uses your logged-in Chrome profile | Usually no (separate profile) | **Yes** — operates the real app |
| Cookie / SSO / extensions | Often lost | **Preserved** |
| `navigator.webdriver` | May be set | **Not applicable** (not in the browser) |
| Platform | Cross-platform | **macOS 13+ only** |
| Platform | Cross-platform | **Windows 10/11 + macOS 13+** |

## Windows quick start

From this checkout on Windows:

```powershell
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\smoke-test-windows.ps1
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\ocu-windows.ps1 apps
powershell.exe -NoProfile -ExecutionPolicy Bypass -File scripts\ocu-windows.ps1 tree --depth 1 --max-nodes 20
```

Cursor is wired by `.cursor/mcp.json`. Codex/plugin MCP uses `.mcp.json`.
Both call `scripts/mcp-server.ps1`, which starts the PowerShell/MS UI Automation
server. macOS configs are kept as `.cursor/mcp.macos.json` and `.mcp.macos.json`.

See [docs/windows-msuia.md](docs/windows-msuia.md) for Windows setup, safety
policy details, and Codex/Cursor JSON snippets.

## Install (recommended)

Expand Down Expand Up @@ -63,12 +86,12 @@ Codex-native manifest: [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json).

Clone or open this repo in Cursor. Project-level config is included:

- MCP: [`.cursor/mcp.json`](.cursor/mcp.json) → `scripts/mcp-server.sh`
- MCP: [`.cursor/mcp.json`](.cursor/mcp.json) -> `scripts/mcp-server.ps1` on Windows. The macOS config is kept as [`.cursor/mcp.macos.json`](.cursor/mcp.macos.json).
- Skill: [`.cursor/skills/open-computer-use/SKILL.md`](.cursor/skills/open-computer-use/SKILL.md)

Restart Cursor or reload MCP. On first connect, `mcp-server.sh` can auto-install the latest release if `ocu` is missing.
Restart Cursor or reload MCP. On macOS, use `.cursor/mcp.macos.json`; `mcp-server.sh` can auto-install the latest release if `ocu` is missing.

For other projects, copy the skill to `~/.cursor/skills/open-computer-use/` and point MCP at `scripts/mcp-server.sh` from your checkout, or at `$(which ocu)` after `./scripts/install.sh`.
For other projects, copy the skill to `~/.cursor/skills/open-computer-use/`. On Windows, point MCP at `scripts/mcp-server.ps1` with `powershell.exe`. On macOS, point MCP at `scripts/mcp-server.sh` from your checkout, or at `$(which ocu)` after `./scripts/install.sh`.

See [examples/plugin-install.md](examples/plugin-install.md) for details and troubleshooting.

Expand Down
Loading