Skip to content

Latest commit

 

History

History
223 lines (153 loc) · 16.6 KB

CLAUDE.md

File metadata and controls

223 lines (153 loc) · 16.6 KB

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

MaaMCP is an MCP (Model Context Protocol) server that exposes MaaFramework's automation capabilities to AI assistants. It provides Android device control via ADB and Windows desktop automation via window handles.

Development Commands

# Install dependencies in development mode
pip install -e .

# Run MCP server (standard serial mode)
maa-mcp
# or
python -m maa_mcp

# Run MCP server (pipeline mode with background screenshot thread)
maa-mcp-server
# or
python -m maa_mcp.pipeline_server

# Run tests
pytest tests/ -v
pytest tests/test_basic.py -v  # run specific file

Architecture

Entry Points

The package has multiple entry points defined in pyproject.toml:

  • maa-mcp / maa_mcp: Standard MCP server (main.py)
  • maa-mcp-server / maa_mcp_server: Pipeline server with multi-threaded background monitoring (pipeline_server.py)

Core Components

  • core.py: Creates the FastMCP server instance, global registries (object_registry, controller_info_registry), and ControllerInfo dataclass
  • registry.py: ObjectRegistry class for managing controller instances by ID
  • paths.py: Cross-platform data directory management using platformdirs

Module Responsibilities

Module Purpose
adb.py ADB device discovery (find_adb_device_list) and connection (connect_adb_device)
win32.py Windows window discovery (find_window_list) and connection (connect_window)
vision.py Screen capture (screencap) and OCR recognition (ocr)
control.py Input operations: click, double_click, swipe, input_text, click_key, keyboard_shortcut, scroll
resource.py Global Resource/Tasker cache; multi-bundle resource path loading; pipeline node management
download.py OCR model file download utilities
agent_supervisor.py Agent 子进程生命周期管理(MFAAvalonia 风格):从 interface.json 读 agent 配置、TCP 模式启子进程、Windows JobObject 绑定、stdout/stderr 流式日志、start / stop / shutdown_all API(设计文档)
pipeline_tools.py Pipeline protocol docs, single/multi-file load_pipeline & run_pipeline, save_pipeline, clear_pipeline_resources, stop_pipeline(原子地关 Tasker / Agent 子进程 / OCR 循环)
pipeline/ Pipeline mode state management and logging

Two Operation Modes

  1. Serial Mode: Synchronous execution where each operation waits for the previous to complete
  2. Pipeline Mode: Multi-threaded mode where a background thread continuously captures screenshots and caches them in a queue for the main thread to process decisions

Controller Pattern

All device/window control flows through:

  1. Discovery functions return device/window identifiers
  2. Connection functions create AdbController or Win32Controller instances (from maafw) and register them in object_registry
  3. Operations use controller_id to look up the controller in object_registry
  4. controller_info_registry stores metadata (controller type, connection params) for each controller_id

Key Dependencies

  • maafw>=5.2.6: Core automation framework (MaaFramework)
  • fastmcp>=2.0.0: MCP server framework
  • opencv-python>=4.0.0: Image processing for screenshots
  • loguru>=0.7.0: Logging
  • platformdirs>=4.0.0: Cross-platform paths

Multi-File Pipeline Loading

run_pipeline(controller_id, pipeline_path) accepts either a single path (str) or a list of paths (list[str]). When multiple files are provided, nodes from all files are merged into the same global node table; node next references resolve in the merged namespace (cross-file references work).

Conflict handling:

  • Default on_conflict="strict": any node name conflict raises an error and writes nothing to the Resource
  • on_conflict="overwrite": later file's entire node replaces earlier file's (MaaFramework default); conflicts are reported in result.warnings

Loading is atomic: all pre-validation (file existence, JSON validity, conflict detection) must pass before a single Resource.override_pipeline(merged) call writes to the Resource. Failures leave the Resource unchanged.

Once loaded, pipeline nodes persist in the Resource until clear_pipeline_resources() is called or the process exits.

JSONC Support (issue #38)

MaaFramework's own pipeline parser (meojson json::parsec) accepts JSONC, so MaaMCP must too — otherwise the same file loads in MaaFramework but fails in MaaMCP.

jsonc.py re-implements that semantics in Python (zero new dependencies):

  • Allowed: // line comments, /* */ block comments, trailing commas in objects/arrays (a comma in an empty container — {,} / [,] — is still a syntax error, same as MaaFramework)
  • Rejected (same as MaaFramework): unquoted keys, single-quoted strings, NaN / Infinity, duplicate keys, lone surrogates (e.g. "\ud800")

Where it applies: pipeline files read by load_pipeline / run_pipeline / benchmark_node, the pipeline_json string passed to save_pipeline (output is still normalized standard JSON — comments are dropped), and interface.json read by the agent supervisor.

Implementation note: comments are replaced with equal-length whitespace (newlines preserved), then trailing commas are stripped, then the text goes to stdlib json.loads. Byte offsets are preserved, so stdlib's line/column error positions apply directly to the original file (a leading UTF-8 BOM shifts them by one character, since the BOM is stripped before parsing).

Three deliberate deviations from meojson:

  1. Numbers that overflow to inf (1e999) are rejected — MaaMCP re-serializes the parsed result into strict JSON for Resource.override_pipeline, so meojson's literal round-trip would write back the invalid literal Infinity.
  2. loads accepts any top-level value while meojson only accepts objects/arrays (callers still require a non-empty object).
  3. Trailing residue after the top-level value is reported as an error. meojson's skip_whitespace conflates "end of input" with "malformed comment start", so it silently accepts and discards a trailing /x or an unterminated block comment; MaaFramework inherits that. This direction is stricter — MaaMCP never accepts a file MaaFramework rejects. An unterminated comment inside the document is rejected by both.

interface.json caveat: MaaMCP tolerates comments there, and so does MFAAvalonia (Newtonsoft.Json skips comments). But MAAGC's own agent/main.py reads that file with stdlib json.load to detect the version — a commented interface.json makes it log an exception and fall back to version "unknown" (it only decides dev-mode vs. venv setup). Keep interface.json comment-free if you rely on MAAGC's version detection.

Agent Subprocess Lifecycle (Custom Action Support)

For pipelines containing action: "Custom" nodes (e.g. MAAGC's TaskProcessor / YearlyTaskProcessor), MaaMCP spawns a Python agent subprocess to host the custom callbacks, following the MFAAvalonia pattern.

How it works:

  1. When run_pipeline is called with a resource_path, the supervisor (agent_supervisor.py) looks up the chain for an interface.json containing an agent block.
  2. If found, it generates a random 8-char identifier, creates an AgentClient.create_tcp(0) (OS-assigned port → identifier "127.0.0.1:<port>"), and binds it to the Resource + Controller + Tasker.
  3. The agent Python subprocess is launched with the identifier appended as the last CLI argument (MAAGC agent/main.py:382 reads it via sys.argv[-1]).
  4. On Windows, the subprocess is bound to a JobObject with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE so a taskkill /F on MaaMCP automatically reaps the agent. On POSIX, start_new_session=True is used plus an atexit fallback.

Lifecycle contract:

  • run_pipeline auto-starts the agent when start_agent=True (default).
  • stop_pipeline(controller_id) atomically stops: OCR loop → tasker.stop() → AgentClient.disconnect() → subprocess terminate() (3s grace) → kill() → registry cleanup.
  • atexit registers agent_supervisor.shutdown_all() as a final safety net.

The agent config schema (read from interface.json):

{
  "agent": {
    "child_exec": "python",
    "child_args": ["-u", "agent/main.py"],
    "identifier": "optional-fixed-id",
    "timeout": 30,
    "auto_start": true
  }
}

Required MCP tool hygiene: The AI must call stop_pipeline(controller_id) after running a pipeline that auto-started an agent. Otherwise the agent subprocess remains running in the background, holding the TCP port, CPU and memory. JobObject on Windows is a safety net but does not free MaaMCP-side resources (TCP socket, registry entries, loguru file handles).

See docs/research/custom-action-support.md for the full design rationale and reference to MFAAvalonia's AgentHelper.cs.

Data Storage

OCR models and screenshots are stored in platform-specific directories:

  • Windows: C:\Users\<user>\AppData\Local\MaaMCP\
  • macOS: ~/Library/Application Support/MaaMCP/
  • Linux: ~/.local/share/MaaMCP/

AI Working Notes

Notes for AI agents (Claude Code, etc.) working in this project. Tool-level details live in the tool descriptions themselves; this section captures cross-cutting patterns.

Tool call pace baseline (one cycle: click → verify → input)

  • Best: 5–10 s/step
  • Normal: 10–20 s/step
  • Anomaly: >30 s/step → reflect: am I over-explaining, re-verifying, or waiting on the wrong thing?

Speedup tactics (in order of impact):

  1. Batch parallel tool calls in one round
  2. Skip intermediate verification (click → input without screencap/ocr between)
  3. Don't re-state what's already obvious; one-line conclusion per action
  4. When uncertain, ask the user, don't self-argue for 5 s

Testing methodology checklist

Before drawing any conclusion from a tool call, verify:

  • "What am I measuring + which variable am I controlling?" — defined before running
  • Repeated at least 2–3 times; single pass never concludes
  • Verified with independent signal (e.g. screencap + ocr), not by guessing "should be here"
  • Between each step, asked "did my prior step contaminate this one?"
  • Interpreted results counterfactually: "could the dropdown have appeared without my click?" / "could the text have been there from a prior test?"
  • "API returned True" ≠ "action took effect visually" — never trust API alone for visual state

MaaMCP tool behavior notes (project-specific)

  • ocr / screencap accept region=(x,y,w,h): regions use full controller screenshot coordinates. OCR uses JOCR(roi=...); the framework restores the origin automatically. Do not repeat roi in roi_offset.
  • Shared coordinate basis: Win32/ADB default to a 720-pixel short side. Connection parameter target_short_side accepts another positive integer or None (native size). screencap defaults to resolution=None, preserving controller image dimensions. OCR, actions, and Pipelines use that controller's coordinates. Non-720 modes do not automatically adapt existing 720p templates or ROIs.
  • Locating targets after cropping/resizing: screencap(..., include_metadata=True) returns path, coordinate_size, image_size, and image_to_controller. Map image point (u,v) to (u*sx+ox,v*sy+oy) and round before acting. The default return remains a path string; metadata's path can be passed to save_captured_image.
  • Win32 click on Chromium/Electron windows can silent-fail — PostMessage mouse events are dropped, API returns True but no visual change. Keyboard events (post_input_text, post_key_*) still work because Chrome's keyboard handler is more lenient. Tool description has the full caveat.
  • input_text requires target focus — it sends keyboard events to the focused element. If click failed to focus, input_text lands elsewhere. Sequence: click(target) → input_text(text) in immediate succession, no screencap/ocr between.
  • Controller has a lifecycle — system cross-day, Chrome restart, or sleep/wake invalidates controller_id. Symptom: all operations return None/False silently. Fix: re-call find_window_list() + connect_window().
  • Don't conflate "test contamination" with "tool bug" — if a tool worked once and then fails, check the test state before assuming the tool is broken. Window switching, manual user interaction between calls, and stale controllers are common contamination sources.
  • save_captured_image writes into project bundle, not MaaMCP data dir — destination is <bundle_root>/image/<subcategory>/<name>.png (the path TemplateMatch's template field reads from). bundle_root is the directory passed to Resource.post_bundle(). For MAAGC it's assets/resource/base/; for MaaFramework sample it's <repo>/sample/resource. Default overwrite=False protects existing templates — pass True explicitly when updating.
  • benchmark_node measures wall-clock, not per-node timing — returned latency_ms is full post_task → TaskDetail time, including entry recognition overhead (~50-200ms). For per-node estimate subtract that baseline. mean_score=None with successes=0 means the node never hit — threshold/ROI/template mismatch.
  • run_pipeline / benchmark_node accept per-run pipeline_override — field-level node overrides passed to post_task (same mechanism as interface.json's pipeline_override): tighten roi, tweak expected/threshold/timeout for single-node verification without editing pipeline files. Applies to one run only — the Resource is untouched. Node names not present in the loaded files show up in warnings (typo'd names silently no-op, so check them). benchmark_node rejects overriding next on entry/target (would break its isolation chain).
  • run_pipeline supports tool-side timeout_seconds — polls the task and calls post_stop() on expiry, returning status="timeout" + partial node details instead of blocking the MCP call indefinitely (a node that never matches burns its own timeout, 20s by default). Note: MaaFramework marks a stopped task's own status as succeeded — the tool reports "timeout" explicitly instead of trusting it. Recommended 5-15s for single-node verification; None (default) keeps the old unbounded behavior.

Pipeline node tuning loop

When a TemplateMatch / OCR / ColorMatch node isn't reliable, iterate this loop (issue #36 item #4):

  1. screencap(cid) → Read source frame, visually locate target region
  2. screencap(cid, region=(x, y, w, h)) → crop just that region
  3. Read cropped image → confirm it's the intended element
  4. save_captured_image(cropped_path, bundle_root, subcategory, name) → promote it to a TemplateMatch template
  5. In pipeline JSON: "recognition": "TemplateMatch", "template": "<subcategory>/<name>.png"
  6. benchmark_node(cid, pipeline_path, node=<name>, iterations=10..50) → inspect mean_score, latency_ms, all_results_samples
  7. If mean_score < 0.85 or successes < iterations: tighten ROI (smaller region), raise threshold, or refresh the template with a fresh capture — try candidate values via pipeline_override={"<name>": {"roi": [...], "threshold": ...}} first (no file edits), then write the winning values back into the pipeline JSON
  8. Repeat 2-7 until stable

For MaaMCP-side pipeline infra testing, see tests/test_dbg_pipeline.py (gated by @pytest.mark.integration; skips if MaaDbgControlUnit DLL isn't shipped).

Stress tests are opt-in — never auto-pick a window

tests/test_performance_stress.py drives real devices/windows (clicks, swipes, typed text, key presses). It is skipped unless MAA_STRESS=1 is set, and the target must be named explicitly via MAA_STRESS_DEVICE / MAA_STRESS_WINDOW.

Do not "fix" it by picking find_window_list()[0]: the first desktop window is usually the user's editor (VS Code), and the stress test's synthetic clicks/keystrokes land on it — this closed VS Code during a plain pytest tests/ run. This mirrors the rule in win32.py: never auto-select a window when several are available.

Localization

Rule: When updating this file, always sync changes to CLAUDE_CN.md