Skip to content

Latest commit

 

History

History
435 lines (321 loc) · 17.5 KB

README_EN.md

File metadata and controls

435 lines (321 loc) · 17.5 KB

LOGO

MaaMCP

license activity stars
MaaFramework PyPI

MCP Server based on MaaFramework Providing Android device and Windows desktop automation capabilities for AI assistants

English | 中文


Introduction

MaaMCP is a Model Context Protocol server that exposes MaaFramework's powerful automation capabilities through standardized MCP interfaces to AI assistants (like Claude). With this server, AI assistants can:

  • 🤖 Android Automation - Connect and control Android devices/emulators via ADB
  • 🖥️ Windows Automation - Control Windows desktop applications
    • 🎯 Background Operation - Screenshots and controls on Windows run in the background without occupying your mouse or keyboard, allowing you to continue using your computer for other tasks
  • 🔗 Multi-Device Coordination - Control multiple devices/windows simultaneously for cross-device automation
  • ⚡ Dual-Mode Operation - Serial mode (synchronous execution) and Pipeline mode (background continuous screenshots), adapting to different scenarios
  • 👁️ Smart Recognition - Use OCR to recognize on-screen text
  • 🎯 Precise Operations - Execute clicks, swipes, text input, key presses, and more
  • 📸 Screenshots - Capture real-time screenshots for visual analysis

Talk is cheap, see: 🎞️ Bilibili Video Demo

Features

🔍 Device Discovery & Connection

  • find_adb_device_list - Scan available ADB devices
  • find_window_list - Scan available Windows windows
  • connect_adb_device - Connect to Android device
  • connect_window - Connect to Windows window

👀 Screen Recognition

  • ocr - Optical Character Recognition (efficient, recommended for priority use, OCR model auto-downloads on first use)
  • screencap - Screenshot capture (use as needed, high token cost)

Screenshot Coordinates

Win32 and ADB connections default to target_short_side=720, preserving aspect ratio. OCR regions and boxes, clicks, swipes, and Pipelines all use the controller's full screenshot coordinates, which may differ from physical device coordinates. screencap now defaults to resolution=None, preserving that image size so full-image coordinates can be used directly.

Pass a positive target_short_side, such as 1080, when connecting, or None for native size. This changes the coordinate basis of the entire controller: templates and Pipeline ROIs must use the corresponding scale. Keep the default for existing 720p resources. Callers that rely on the previous Win32 1080p coordinates can explicitly select 1080.

screencap still returns a path string by default. When locating targets in cropped or explicitly resized images, pass include_metadata=True to receive:

Field Meaning
path Absolute image path, also accepted by save_captured_image
controller_id / coordinate_system Owning controller and coordinate space (controller)
coordinate_size Full controller screenshot [width, height]
image_size Saved image [width, height]
image_to_controller.scale Image-to-controller scale [sx, sy]
image_to_controller.offset Actual crop origin in controller coordinates [ox, oy]

Map an image point (u, v) to (round(u*sx+ox), round(v*sy+oy)) before clicking. The transform includes actual resized dimensions, rounding, and crop boundary clamping. It only describes this capture; take another screenshot after the window size or layout changes. Explicit resolution only affects the saved image, not controller coordinates, and cannot restore detail lost in an earlier downsampling step.

🎮 Device Control

  • click - Click at coordinates (supports multi-touch/mouse button selection, long press)
    • On Windows, supports mouse button selection: left, right, middle
  • double_click - Double click at coordinates
  • swipe - Swipe gesture (preferred for scrolling/paging on Android)
  • input_text - Input text
  • click_key - Key press (supports long press)
    • On Android, simulates system keys: Back(4), Home(3), Menu(82), Volume keys, etc.
    • On Windows, supports virtual key codes: Enter(13), ESC(27), Arrow keys, etc.
  • keyboard_shortcut - Keyboard shortcuts
    • Supports key combinations: Ctrl+C, Ctrl+V, Alt+Tab, etc.
  • scroll - Mouse wheel (Windows only)

⚡ Pipeline Mode (Multi-threaded Background Monitoring)

  • start_pipeline - Start background monitoring pipeline, continuously captures screenshots and caches image paths
  • stop_pipeline - Stop pipeline
  • get_new_messages - Get new screenshot paths cached by pipeline
  • get_pipeline_status - Get pipeline running status

📝 Pipeline Generation & Execution

  • get_pipeline_protocol - Get Pipeline protocol documentation
  • save_pipeline - Save Pipeline JSON to file (supports creating and updating)
  • load_pipeline - Load an existing Pipeline file
  • run_pipeline - Run Pipeline and return execution results (single/multi-file, auto-start Custom action agent, per-run pipeline_override, tool-side timeout_seconds)

🛑 Pipeline Termination

  • stop_pipeline - Atomically stop Tasker / Agent subprocess / OCR loop to prevent orphan processes

Quick Start

Installation

Option 1: Install via pip (Recommended)

pip install maa-mcp

Option 2: Install from source

  1. Clone the repository

    git clone https://github.com/MistEO/MaaMCP.git
    cd MaaMCP
  2. Install Python dependencies

    pip install -e .

Configure Clients

Add an MCP server to software such as Cursor:

{
  "mcpServers": {
    "MaaMCP": {
      "command": "maa-mcp"
    }
  }
}

OR

Add an MCP command to software such as Cherry Studio:

maa-mcp

Usage Examples

After configuration, you can use it:

Android Automation Example:

Please use the MaaMCP tools to connect to my Android device, open Meituan, and help me order a Chinese meal (one portion) around 20 RMB.

Windows Automation Example:

Please use the MaaMCP tools to show me how to add a rotation animation effect to the current PPT slide, and demonstrate the steps.

Pipeline Generation Example:

Please use MaaMCP tools to connect to my device, help me open Settings, go to Display settings, and adjust brightness to 50%.
After completing the operations, generate a Pipeline JSON for this workflow so it can be run directly later.

MaaMCP will automatically:

  1. Scan available devices/windows
  2. Establish connection
  3. Auto-download and load OCR resources (on first use)
  4. Execute recognition and operation tasks

Workflow

MaaMCP follows a streamlined operational workflow with multi-device/window coordination support and two operation modes:

graph LR
    A[Scan Devices] --> B[Establish Connection]
    B --> C1[Serial Mode]
    B --> C2[Pipeline Mode]
    C1 --> D[Execute Automation]
    C2 --> D
Loading
  1. Scan - Use find_adb_device_list or find_window_list
  2. Connect - Use connect_adb_device or connect_window (can connect multiple devices/windows, each gets a unique controller ID)
  3. Operate - Execute OCR, click, swipe, etc. on multiple devices/windows by specifying different controller IDs (OCR resources auto-download on first use)

Dual-Mode Operation

MaaMCP supports two operation modes, allowing flexible selection based on task requirements:

Serial Mode (Default)

Traditional synchronous execution, completing one instruction before executing the next:

OCR Recognition → Analyze Results → Execute Operation → OCR Recognition → ...

Suitable for: Simple tasks, scenarios with low real-time requirements

Pipeline Mode (Multi-threaded Background Monitoring)

Multi-threaded asynchronous execution, background thread continuously captures screen information while main thread focuses on decision-making and operations:

graph LR
    subgraph Background Thread
        S1[Continuous Screenshots] --> S2[Cache Image Paths]
        S2 --> S3[Push to Message Queue]
        S3 --> S1
    end
    subgraph Main Thread
        M1[Get Screenshot Path] --> M2[Visual Analysis]
        M2 --> M3[Decide if OCR Needed]
        M3 --> M4[Execute Operation]
        M4 --> M1
    end
Loading

Workflow:

  1. Start Pipeline - Call start_pipeline(controller_id) to start background monitoring
  2. Get Screenshots - Call get_pipeline_status() to check status, get_new_messages() to get screenshot paths
  3. Analyze & Execute - Read images for visual analysis, call OCR as needed, execute clicks and other operations
  4. Stop Pipeline - Call stop_pipeline() to release resources when task is complete

Advantages:

  • Continuous background screenshots, AI can directly view complete screen for decision-making
  • AI can decide whether OCR is needed and which specific regions to OCR based on image content
  • Supports high-frequency screen monitoring, never misses interface changes
  • Suitable for real-time automation tasks requiring fast response
  • Message queue mechanism supports asynchronous processing

Usage Example:

Please use MaaMCP tools to connect to my device, use pipeline mode to monitor the screen, and automatically click confirm when a specific popup appears.

Pipeline Generation

MaaMCP supports AI converting executed operations into MaaFramework Pipeline JSON format, enabling operate once, reuse infinitely.

How It Works

graph LR
    A[AI Executes Operations] --> B[Operations Completed]
    B --> C[AI Reads Pipeline Docs]
    C --> D[AI Intelligently Generates Pipeline]
    D --> E[Save JSON File]
    E --> F[Run Validation]
    F --> G{Success?}
    G -->|Yes| H[Done]
    G -->|No| I[Analyze Failure]
    I --> J[Modify Pipeline]
    J --> F
Loading
  1. Execute Operations - AI performs OCR, clicks, swipes, and other automation operations normally
  2. Get Documentation - Call get_pipeline_protocol to get the Pipeline protocol specification
  3. Intelligent Generation - AI converts valid operations into Pipeline JSON based on the documentation
  4. Save File - Call save_pipeline to save the generated Pipeline
  5. Run Validation - Call run_pipeline to verify the Pipeline works correctly
  6. Iterative Optimization - Analyze failures and modify Pipeline until successful

Advantages of Intelligent Generation

Unlike mechanical recording, AI intelligent generation offers these advantages:

  • Only Keeps Successful Paths: If multiple paths were tried during operation (e.g., first entering Menu A without finding the target, then returning and entering Menu B to find it), AI will only keep the final successful path, removing failed attempts
  • Understands Operation Intent: AI can understand the purpose of each operation and generate semantically clear node names
  • Optimizes Recognition Conditions: Intelligently sets recognition regions and matching conditions based on OCR results
  • Validation & Iteration: Discovers issues through run validation, automatically fixes and enhances robustness

Validation & Iterative Optimization

After Pipeline generation, AI automatically validates and optimizes:

  1. Run Validation - Execute Pipeline to check if it succeeds
  2. Failure Analysis - If failed, analyze which node failed and why
  3. Intelligent Fixes - Common optimization techniques:
    • Add alternative recognition nodes (add multiple candidates in next list)
    • Relax OCR matching conditions (use regex or partial matching)
    • Adjust roi recognition regions
    • Add wait time (post_delay)
    • Add intermediate state detection nodes
  4. Re-validate - Run again after modifications until it consistently succeeds

If the Pipeline logic itself needs adjustment, AI can re-execute automation operations and combine old and new experiences to generate a more robust Pipeline.

Fast Single-Node Verification (pipeline_override + timeout_seconds)

When debugging a single node, there is no need to repeatedly edit the pipeline file, suffer slow full-screen recognition, or let the call block forever:

run_pipeline(
    controller_id,
    "main.json",
    entry="ClickSettings",                  # enter from the node under test
    start_agent=False,                      # recognition only, no CustomAction chain
    pipeline_override={                     # per-run only: no file edits, Resource untouched
        "ClickSettings": {"roi": [520, 20, 200, 80], "timeout": 3000}
    },
    timeout_seconds=10,                     # tool-side cap: auto post_stop after 10s
)
  • pipeline_override uses the same mechanism as interface.json's field: field-level merge into loaded nodes; any field (expected / threshold / enabled / ...) can be overridden — and the tighter the ROI, the faster each recognition
  • On timeout the result carries status="timeout" plus the nodes executed before the stop, showing exactly where it hung
  • benchmark_node accepts the same pipeline_override, making tuning loops file-edit-free

Example Output

{
  "StartTask": {
    "recognition": "DirectHit",
    "action": "DoNothing",
    "next": ["ClickSettings"]
  },
  "ClickSettings": {
    "recognition": "OCR",
    "expected": "Settings",
    "action": "Click",
    "next": ["EnterDisplay"]
  },
  "EnterDisplay": {
    "recognition": "OCR",
    "expected": "Display",
    "action": "Click",
    "next": ["AdjustBrightness"]
  },
  "AdjustBrightness": {
    "recognition": "OCR",
    "expected": "Brightness",
    "action": "Swipe",
    "begin": [200, 500],
    "end": [400, 500],
    "duration": 200
  }
}

Notes

📌 Windows Automation Limitations:

  • Some games or applications with anti-cheat mechanisms may block background control operations
  • If the target application runs with administrator privileges, MaaMCP must also be launched with administrator privileges
  • Minimized windows are not supported; please keep the target window in a non-minimized state
  • If the default background screenshot/input methods are unavailable (e.g., empty screenshots, unresponsive operations), the AI assistant may attempt to switch to foreground methods, which will occupy the mouse and keyboard

Platform Support

Feature Windows macOS Linux
MCP server (maa-mcp) Yes Yes Yes
Pipeline mode (maa-mcp-server) Yes Yes Yes
find_adb_device_list / connect_adb_device Yes Yes Yes
find_window_list / connect_window Yes No No
Background screenshot / mouse / keyboard Yes No No

Linux notes:

  • ADB-based Android control works on Linux.
  • Win32 desktop window control is not supported on Linux/macOS — find_window_list() and connect_window() will return None / raise on non-Windows platforms.
  • If you see ANSI color escape sequences (e.g. [31m...) leaking into MCP stdio, that is a known interaction with some MCP clients on Linux. Log files are written to ~/.local/share/MaaMCP/logs/ and are unaffected.

FAQ

FAQ

OCR recognition fails with "Failed to load det or rec" or prompts that resources do not exist

On first use, OCR model files will be automatically downloaded. However, download failures may occur. Please check the data directory:

  • Windows: C:\Users\<username>\AppData\Local\MaaMCP\resource\model\ocr\
  • macOS: ~/Library/Application Support/MaaMCP/resource/model/ocr/
  • Linux: ~/.local/share/MaaMCP/resource/model/ocr/
  1. Check if model files exist in the above directory (det.onnx, rec.onnx, keys.txt)
  2. Check for resource download errors in model/download.log
  3. Manually run python -c "from maa_mcp.download import download_and_extract_ocr; download_and_extract_ocr()" to retry downloading

LM Studio says 'maa-mcp' is not recognized as an internal or external command

When you configure the MCP server in LM Studio, it needs to find the maa-mcp executable on PATH. On Windows, pip install puts scripts into <Python install dir>\Scripts\ (e.g. C:\Users\<user>\AppData\Local\Programs\Python\Python310\Scripts\), which is often not on PATH by default. Fix:

# Add Scripts dir to your user PATH (permanent)
setx PATH "%PATH%;%LOCALAPPDATA%\Programs\Python\Python310\Scripts"

Then restart LM Studio. To verify: where maa-mcp in a new terminal should return the script path. See #6.

About Issues

When filing an issue, please provide log files. The log file path is as follows:

  • Windows: C:\Users\<username>\AppData\Local\MaaMCP\debug\maa.log
  • macOS: ~/Library/Application Support/MaaMCP/debug/maa.log
  • Linux: ~/.local/share/MaaMCP/debug/maa.log

License

This project is licensed under GNU AGPL v3.

Acknowledgments