Skip to content

README_EN.md

Latest commit

 

History

History
400 lines (276 loc) · 16.3 KB

README_EN.md

File metadata and controls

400 lines (276 loc) · 16.3 KB
Raw
autoglm-gui

AutoGLM-GUI

Modern Web GUI for AutoGLM Phone Agent - AI-Powered Android Device Automation Made Simple

🎉 Android 11+ devices now support fully wireless pairing via QR code, no cable required! 🎉

Python License PyPI
Join our Discussion Group

中文文档

✨ Features

  • Fully Wireless Pairing - 🆕 Android 11+ QR code pairing, no cable needed
  • Multi-Device Control - Manage and control multiple Android devices simultaneously with isolated states
  • Conversational Task Management - Control Android devices through chat interface
  • Workflow System - 🆕 Predefine common tasks for one-click execution, with full CRUD support
  • Real-time Screen Preview - Low-latency scrcpy video streaming to monitor device operations
  • Direct Device Control - Click and swipe directly on live video with precise coordinate mapping and visual feedback
  • Zero-Configuration Deployment - Works with any OpenAI-compatible LLM API
  • Deep ADB Integration - Direct device control via Android Debug Bridge (USB & WiFi support)
  • Modular Interface - Clean sidebar + device panel design with clear separation of concerns

📥 Quick Download

One-Click Desktop Version (No Environment Setup Required):

Platform Download Link Notes
🪟 Windows (x64) 📦 Download Portable EXE For Windows 10/11, no installation needed
🍎 macOS (Apple Silicon) 📦 Download DMG For M-series Macs
🐧 Linux (x64) 📦 Download AppImage | deb | tar.gz Universal format for major distributions

Usage Instructions:

  • Windows: Download and double-click the .exe file to run, no installation required
  • macOS: Download and double-click the .dmg file, drag to Applications folder. First launch may require allowing in System Settings → Privacy & Security
  • Linux:
    • AppImage (Recommended): Download, add execute permission chmod +x AutoGLM*.AppImage, then run directly
    • deb: For Debian/Ubuntu systems, install with sudo dpkg -i autoglm*.deb
    • tar.gz: Portable version, extract and run ./AutoGLM\ GUI/autoglm-gui

💡 Tip: Desktop version includes all dependencies (Python, ADB, etc.), no manual environment setup needed. You'll need to configure model service API on first launch.

Or use Python package (requires Python environment):

# Install via pip (recommended)
pip install autoglm-gui

# Or run without installation using uvx (requires uv installed)
uvx autoglm-gui

📸 Screenshots

Quick jump: Classic Mode · Dual Model (Enhanced) · Layered Agent (Enhanced)

Dual Model Architecture

Decision model (e.g., GLM-4.7) + Vision model (AutoGLM-Phone): the decision model plans and recovers from errors, while the vision model observes the UI and executes actions—better for longer and more complex flows.

  • 🧠 Decision layer: task understanding / step planning / exception recovery
  • 👁️ Execution layer: identify UI elements and click / swipe / type to operate
  • 🔄 Runtime loop: plan → execute → feedback; re-plan when needed

Thinking modes: TURBO (fastest & cheapest for routine flows) / DEEP (most robust for complex tasks) / FAST (quick for lightweight tasks).

Dual Model UI

Layered Agent

Layered Agent is a stricter two-layer setup: the planner layer decomposes the task and reasons over multiple turns; the executor layer focuses on observing and operating the phone. The planner drives the executor via tool calls (visible in UI), making the process more transparent and easier to adjust mid-flight.

Layered Agent UI

Task Started

Task Started

Task Completed

Task Completed

Multi-Device Control

Multi-Device Control

🚀 Quick Start

🎯 Model Service Configuration

AutoGLM-GUI requires an OpenAI-compatible model service. You can:

  • Use officially hosted third-party services:
    • Zhipu BigModel: --base-url https://open.bigmodel.cn/api/paas/v4, --model autoglm-phone, --apikey <your API Key>
    • ModelScope: --base-url https://api-inference.modelscope.cn/v1, --model ZhipuAI/AutoGLM-Phone-9B, --apikey <your API Key>
  • Or self-host: Refer to upstream project's deployment guide to deploy zai-org/AutoGLM-Phone-9B with vLLM/SGLang, then point --base-url to your service.

Examples:

# Using Zhipu BigModel
pip install autoglm-gui
autoglm-gui \
  --base-url https://open.bigmodel.cn/api/paas/v4 \
  --model autoglm-phone \
  --apikey sk-xxxxx

# Using ModelScope
pip install autoglm-gui
autoglm-gui \
  --base-url https://api-inference.modelscope.cn/v1 \
  --model ZhipuAI/AutoGLM-Phone-9B \
  --apikey sk-xxxxx

# Using your self-hosted vLLM/SGLang service
pip install autoglm-gui
autoglm-gui --base-url http://localhost:8000/v1 --model autoglm-phone-9b

Prerequisites

  • Python 3.10+
  • Android device (Android 11+ supports fully wireless pairing, no cable needed)
  • ADB installed and added to system PATH (desktop version includes ADB)
  • An OpenAI-compatible API endpoint

About Device Connection:

  • Android 11+: QR code pairing supported, no cable needed for connection and control
  • Android 10 and below: Initial USB connection required to enable wireless debugging, then cable can be removed

Quick Run (Recommended)

No manual environment setup required, just install and run:

# Install via pip and start
pip install autoglm-gui
autoglm-gui --base-url http://localhost:8080/v1

Or use uvx for installation-free startup with latest version (requires uv installed, installation guide):

uvx autoglm-gui --base-url http://localhost:8080/v1

Traditional Installation

# Install from source
git clone https://github.com/your-repo/AutoGLM-GUI.git
cd AutoGLM-GUI
uv sync

# Build frontend (required)
uv run python scripts/build.py

# Start service
uv run autoglm-gui --base-url http://localhost:8080/v1

After startup, open http://localhost:8000 in your browser to start using!

🔄 Upgrade Guide

Check Current Version

# View installed version
pip show autoglm-gui

# Or use command-line argument
autoglm-gui --version

Upgrade to Latest Version

Using pip:

# Upgrade to latest version
pip install --upgrade autoglm-gui

📖 Usage Guide

Multi-Device Management

AutoGLM-GUI supports controlling multiple Android devices simultaneously:

  1. Device List - Left sidebar automatically shows all connected ADB devices
  2. Device Selection - Click device card to switch to its control panel
  3. Status Indicators - Clearly shows each device's online and initialization status
  4. State Isolation - Each device has independent chat history, config, and video stream

Device Status Indicators:

  • 🟢 Green dot: Device online
  • ⚪ Gray dot: Device offline
  • ✓ Mark: Device initialized

📱 QR Code Wireless Pairing (Android 11+ Recommended)

No cable needed, phone and computer just need to be on the same WiFi network:

  1. Phone Setup:

    • Open Settings → Developer Options → Enable "Wireless Debugging"
    • Keep phone and computer on the same WiFi network

  2. Computer Operations:

    • Click ➕ "Add Wireless Device" button in bottom-left corner
    • Switch to "Pair Device" tab
    • QR code auto-generated, ready for scanning

  3. Phone Scanning:

    • In "Wireless Debugging" page, tap "Pair device with QR code"
    • Scan the QR code displayed on computer
    • After successful pairing, device automatically appears in device list

Features:

  • ✅ No cable needed
  • ✅ One-click QR code pairing
  • ✅ Automatic device discovery and connection
  • ✅ Works with Android 11 and above

AI Automation Mode

  1. Connect Device - Use any method above (QR code pairing recommended for Android 11+)
  2. Select Device - Choose device to control in left sidebar
  3. Initialize - Click "Initialize Device" button to configure Agent
  4. Chat - Describe what you want to do (e.g., "Order a Boba Milk Tea from Meituan")
  5. Observe - Agent executes operations step-by-step, showing thinking process and actions in real-time

🌿 Classic Mode (Single Model / Open AutoGLM)

This is the default Open AutoGLM-Phone experience: a single vision model completes the full loop of understanding → planning → observing → acting.

  • Pros: simplest setup, fastest to get started
  • Best for: clear goals with fewer steps (e.g., open an app, simple navigation)

🧠 Dual Model Mode (Enhanced)

Dual model mode combines a decision model (planning & recovery) with a vision model (UI execution) for better stability and controllability on harder tasks.

Thinking modes:

  • TURBO (recommended): plan an action sequence once, batch-execute; re-plan only on errors
  • DEEP: involve the decision model at every step for maximum robustness
  • FAST: step-by-step decisions with shorter prompts for quicker responses

Config tips:

  • Decision model: use a strong planning/reasoning model (GLM-4.7 / GPT-4 / Claude, etc.)
  • Vision model: use a GUI-capable model (AutoGLM-Phone-9B / autoglm-phone)

🧩 Layered Agent Mode (Enhanced / Experimental)

Layered Agent is a stricter two-layer design: the planner focuses on decomposition and multi-turn reasoning, while the executor focuses on observing and operating the phone.

  • How it works: the planner calls tools (e.g., list_devices() / chat(device_id, message)) to drive the executor; you can see tool calls and results in the UI
  • Granularity: the executor runs small, atomic sub-tasks with a step limit, so the planner can adjust strategy based on feedback
  • Important limitation: the executor doesn’t “take notes” or reliably extract/save text as variables—you must ask it to read what’s on screen when you need information

Manual Control Mode

Besides AI automation, you can also directly control the phone on live screen:

  1. Live Screen - Right side of device panel shows real-time video stream (scrcpy-based)
  2. Click Operations - Click anywhere on the screen, operations sent to phone immediately
  3. Swipe Gestures - Hold and drag mouse to perform swipe operations (scroll wheel supported)
  4. Visual Feedback - Each operation shows ripple animation and success/failure notification
  5. Precise Mapping - Automatically handles screen scaling and coordinate transformation for accurate positioning
  6. Display Modes - Support switching between auto, video stream, and screenshot modes

Workflow Management

Save common tasks as Workflows for one-click execution:

Creating and Managing Workflows

  1. Enter Management Page - Click Workflows icon (📋) in left sidebar
  2. Create New Workflow - Click "New Workflow" button in top-right corner
  3. Fill Information:
    • Name: Give the Workflow a short, memorable name (e.g., "Order Boba Tea")
    • Task Content: Detailed description of task to execute (e.g., "Order a Boba Milk Tea from Meituan, no ice, add pearls")
  4. Save - Click save button

Management Operations:

  • Edit - Click "Edit" button on Workflow card to modify content
  • Delete - Click "Delete" button to remove unwanted Workflows
  • Preview - Workflow card shows preview of first few lines of task content

Quick Workflow Execution

Execute saved Workflows in Chat interface:

  1. Select Device - Ensure target device is selected and initialized
  2. Open Workflow Selector - Click Workflow button (📋 icon) next to input box
  3. Select Task - Click the Workflow you want to execute from the list
  4. Auto-fill - Task content automatically fills into input box
  5. Send and Execute - Click send button to start execution

Use Case Examples:

  • 📱 Daily Tasks: Order food, call taxi, check delivery
  • 🎮 Gaming Operations: Daily check-in, claim rewards
  • 📧 Message Sending: Bulk messages with fixed content
  • 🔄 Repetitive Operations: Periodic maintenance tasks

🛠️ Development Guide

Quick Development

# Backend development (auto-reload)
uv run autoglm-gui --base-url http://localhost:8080/v1 --reload

# Frontend dev server (hot reload)
cd frontend && pnpm dev

Build and Package

# Build frontend only
uv run python scripts/build.py

# Build complete package
uv run python scripts/build.py --pack

🤝 How to Contribute

We warmly welcome community contributions! Whether it's fixing bugs, adding new features, improving documentation, or sharing your experiences, every contribution is valuable to the project.

🎯 Quick Start Contributing

  1. Check the Pinned Issue - 🎯 Start Here: How to Contribute / Claim Tasks / Run Locally
  2. Read the Contribution Guide - See CONTRIBUTING.md for detailed steps
  3. Claim a Task - Comment /assign me on the Issue you're interested in

💡 Ways to Contribute

  • 🐛 Fix Bugs - Look for Issues labeled bug
  • ✨ Add Features - Implement Issues labeled enhancement
  • 📖 Improve Documentation - Fix errors, add explanations, provide examples
  • 🧪 Add Tests - Improve code quality and test coverage
  • 🌍 Translate Docs - Help users in other languages

🏷️ Beginner-Friendly Tasks

If this is your first time contributing to open source, you can start with:

  • Look for Issues labeled good first issue
  • Improve documentation (fix typos, add explanations)
  • Test the software and report your experience

📚 Reference Resources

Document Description
CONTRIBUTING.md Complete contribution guide (setup, workflow, PR standards)
CLAUDE.md Technical architecture documentation (code structure, key details)
Issues Browse and claim tasks

💬 Discussion

  • 💭 Discuss ideas and questions in Issues
  • 🎮 Join our QQ Group
  • 📝 Create a new Issue to report problems or suggest improvements

Thank you to all contributors - you make AutoGLM-GUI better! 🎉

📝 License

Apache License 2.0

License Notice

AutoGLM-GUI is licensed under Apache License 2.0. However, it bundles ADB Keyboard APK (com.android.adbkeyboard), which is licensed under GPL-2.0. The ADB Keyboard component is used as an independent tool and does not affect AutoGLM-GUI's Apache 2.0 license.

See: AutoGLM_GUI/resources/apks/ADBKeyBoard.LICENSE.txt

🙏 Acknowledgments

This project is built upon Open-AutoGLM. Thanks to the zai-org team for their excellent work on AutoGLM.