Cutscene Copilot

Languages: English | 中文

Companion repositories: cutscene_agent · CutsceneProvider

A VS Code GitHub Copilot customization for Unreal Engine cutscene creation. Through custom Agents, Skills, and MCP servers, it enables end-to-end cutscene creation from script to UE Level Sequence.

Features

• Script-Driven Creation — Paste a script and the AI automatically handles character placement, dialogue voiceover, body animation choreography, and camera design
• Voice & Expression Asset Tools — Built-in MCP server stubs. Plug in your own TTS and facial expression generation tools.
• Intelligent Camera Choreography — Select and combine camera shots from template library, supporting Dolly, Orbit, and other camera movements
• Virtual Photographer — Independent sub-Agent that controls the UE editor viewport for real-time composition adjustments

Quick Start

1. Install prerequisites (see "Prerequisites" below)
2. Configure MCP connection: Copy .vscode/mcp.example.json to .vscode/mcp.json and adjust addresses as needed
3. Start creating: Select the cutscene-director Agent in VS Code Copilot Chat and paste your script

Project Structure

.github/
├── agents/
│   ├── cutscene-director.agent.md    # Director Agent: main creation workflow
│   └── photographer.agent.md         # Photographer Agent: editor viewport control
└── skills/
    └── cutscene-creator/
        ├── SKILL.md                   # Creation workflow Skill entry point
        └── references/
            ├── rules-cutscene-workflow.md  # Content structure & creation workflow
            ├── rules-asset-system.md      # Asset query & import
            ├── rules-actor.md             # Character placement rules
            ├── rules-audio.md             # Audio / TTS / facial expression rules
            ├── rules-camera.md            # Camera design rules
            └── script-format.md           # Script format guide
locales/
└── zh/                                # Chinese version (mirrors .github/ structure)
mcp_servers/
└── aigc_asset_tools.py                # AIGCAssetTools MCP server (built-in)
.vscode/
└── mcp.example.json                   # MCP connection config template

Prerequisites

• VS Code with GitHub Copilot extension (Agent mode support required)
• Unreal Engine project with CutsceneProvider plugin installed (provides the MCP server)
• Python 3.10+ for running the AIGCAssetTools MCP service

MCP Servers

This project depends on two MCP servers:

Server	Purpose	Connection
CutsceneProvider	UE Level Sequence operations (characters / cameras / animations / timeline)	HTTP, launched by the UE plugin
AIGCAssetTools	TTS audio generation, facial expression generation, file import bridge	Stdio (script included in this repo)

Install AIGCAssetTools Dependencies

pip install -r mcp_servers/requirements.txt

MCP Connection Configuration

Copy the config template and modify as needed:

cp .vscode/mcp.example.json .vscode/mcp.json

Default configuration (.vscode/mcp.example.json):

{
  "servers": {
    "CutsceneProvider": {
      "url": "http://localhost:8100/mcp",  // UE plugin MCP address, adjust as needed
      "type": "http"
    },
    "AIGCAssetTools": {
      "type": "stdio",
      "command": "python",
      "args": ["${workspaceFolder}/mcp_servers/aigc_asset_tools.py", "mcp",
               "--ue-mcp-url", "http://localhost:8100/mcp"]
    }
  }
}

• CutsceneProvider: Start the CutsceneProvider plugin in the UE editor first; it provides an HTTP MCP service on the specified port
• AIGCAssetTools: Uses stdio mode; VS Code automatically manages the process lifecycle. The --ue-mcp-url argument specifies the CutsceneProvider address, used for bridging features like push_file_to_ue

Usage

Option 1: Director Agent (Recommended)

1. Ensure the CutsceneProvider MCP service is running (enable the plugin in UE editor)
2. Open Copilot Chat in VS Code
3. Select the cutscene-director Agent
4. Paste your script content — the director will automatically complete the full creation workflow

Option 2: Skill Invocation

Use the cutscene-creator Skill in Copilot Chat and follow the guided creation process.

Option 3: Photographer

The Director Agent will automatically delegate to the photographer sub-Agent when composition adjustments are needed.

Creation Workflow Overview

Script Input
  │
  ├─ Step 1: Query Assets → Discover available characters, animations, voice tones, camera templates
  ├─ Step 2: Add Characters → Place characters and adjust orientation
  ├─ Step 3: Generate Audio → TTS voiceover → Facial expressions → Import to UE → Add to timeline
  ├─ Step 4: Body Animation → Match animations by emotion / duration
  ├─ Step 5: Camera Design → Create cameras, apply templates, set up cuts
  └─ Step 6: Review & Adjust → Verify timeline integrity

For detailed creation workflow, tool dependency chains, and rules, refer to the Skill documentation in .github/skills/cutscene-creator/.

Multilingual Support

The default .github/ contains English versions. Chinese versions are available in locales/zh/.github/.

Switch to Chinese: Copy the contents of locales/zh/.github/ over .github/:

cp -r locales/zh/.github/* .github/

Customizing Skills for Your Project

The Skill rules provided in this repo are written for the sample project. Some rules may need adjustment for your project. For example:

• Animation timeline rules: The default rule is "no gaps allowed on animation timeline; brief overlaps are acceptable (UE handles blend transitions automatically)". If your project uses Blueprint-based Idle states, gaps may be acceptable. If your animations are designed to chain seamlessly, overlaps should be avoided. Adjust rules-cutscene-workflow.md accordingly.
• Asset query fields: If your CutsceneProvider uses different asset field structures, the Agent will dynamically discover them at runtime via get_query_instruction — no Skill file changes needed.
• Camera templates: The Agent dynamically queries via get_available_camera_templates; custom templates will be automatically available.

We recommend reading the rule files in .github/skills/cutscene-creator/references/ and adjusting them to your project's needs.

Citation

If you find this work useful, please cite our paper:

@article{he2026cutscene,
  title={Cutscene Agent: An LLM Agent Framework for Automated 3D Cutscene Generation},
  author={He, Lanshan and Pang, Haozhou and Gan, Qi and Shen, Xin and Zhang, Ziwei and Liu, Yibo and Fang, Gang and Liu, Bo and Sheng, Kai and Zeng, Shengfeng and Li, Chaofan and Hui, Zhen and Zhou, Keer and Zhou, Lan and Dai, Shujun},
  journal={arXiv preprint arXiv:2604.25318},
  year={2026}
}

License

This project is licensed under the MIT License. See the LICENSE file for details.