UE Agent 2 - Professional Greybox Scene Generator for Unreal Engine 5

A production-ready pipeline that transforms natural language prompts into professional greybox UE5 scenes with automated material systems, proper lighting, and clean architectural workflows.

🎯 Overview

This project converts natural language descriptions into professional greybox scenes in Unreal Engine 5. It uses GPT-4 for intelligent prompt parsing and generates industry-standard greybox environments with color-coded materials, optimized lighting, and proper UE5 workflow integration.

Target Demo Prompt: "Give me a cosy loft with a skylight and put a rocket-shaped lamp on the bedside table."

✨ Key Features

• 🎨 Professional Greybox Materials: Automated color-coded material system

  • Walls: 0.7 grey for clean architectural definition
  • Floors: 0.5 grey for ground plane clarity
  • Ceilings: 0.8 grey for overhead visibility
  • Objects: 0.6 grey for furniture and props
  • Doors: 0.4 grey for portal identification
  • Windows: 0.9 grey for light source marking

• 💡 Optimized Lighting Setup: Professional 3-point lighting system

  • Single DirectionalLight as primary sun source
  • PointLight for fill lighting and depth
  • SkyLight for ambient environment illumination

• 🏗️ Clean UE5 Workflow: Industry-standard naming and organization

  • Professional "Greybox_" naming conventions
  • Proper object hierarchy and positioning
  • Ready for level design iteration and asset replacement

🏗️ Architecture

1. Prompt Parser (src/prompt_parser.py) - Uses OpenAI GPT-4 function calling to convert text into structured JSON
2. Greybox Scene Builder (src/ue5_scene_builder.py) - Headless UE5 Python script with automated material system and professional lighting
3. Demo Pipeline (scripts/run_demo.sh/.bat) - One-click execution from prompt to production-ready greybox scene

🚀 Quick Start

Prerequisites

• Unreal Engine 5.3+ with Python API enabled
• Python 3.8+ with virtual environment
• OpenAI API key for GPT-4 function calling
• Windows or macOS (cross-platform compatible)

Installation

1. Clone Repository

   git clone <repository-url>
   cd UEagent2

2. Setup Python Environment

   python -m venv venv
   source venv/bin/activate  # On Windows: venv\Scripts\activate
   pip install -r requirements.txt

3. Configure OpenAI API

   # Create .env file with your OpenAI API key
   echo "OPENAI_API_KEY=your-api-key-here" > .env

4. Setup Unreal Engine 5.3+

   • Install Unreal Engine 5.3 or later from Epic Games Launcher
   • Enable Python API in UE5:
     • Go to Edit → Plugins
     • Search for "Python Editor Script Plugin"
     • Enable the plugin and restart UE5
   • Create or open a UE5 project where scenes will be generated
   • Ensure the project has a /Game/ content directory

Basic Usage

1. Run the Demo Script (Recommended)

   ./scripts/run_demo.sh  # On macOS/Linux
   # or
   scripts\run_demo.bat   # On Windows

2. Enter Your Prompt

   Create a modern bedroom with a king-size bed, bedside table with a rocket-shaped lamp, 
   and natural lighting from a skylight
   

3. Generated Output

   • Scene JSON: Structured scene description in scene.json
   • UE5 Scene: Professional greybox scene saved as /Game/GeneratedScene.umap
   • Materials: Automatically applied greybox materials with industry-standard color palette

Alternative: Direct Python Execution

If you prefer to run the components individually:

python src/prompt_parser.py

How to Replace Prompts and Regenerate

Method 1: Using Demo Script (Recommended)

1. Run the demo script: ./scripts/run_demo.sh (or scripts\run_demo.bat on Windows)
2. Enter your new prompt when prompted
3. The system will automatically parse and generate a new scene

Method 2: Interactive Python Mode

1. Run the generator directly: python src/prompt_parser.py
2. Enter your new prompt when prompted
3. The system will automatically parse and generate a new scene

Method 3: Direct Script Modification

1. Open src/prompt_parser.py
2. Find the prompt input section and replace with your desired prompt:

   # Replace this line with your custom prompt
   prompt = "Your custom scene description here"

3. Run via demo script: ./scripts/run_demo.sh

Method 4: Programmatic Usage

from src.prompt_parser import PromptParser
from src.ue5_scene_builder import UE5SceneBuilder

# Parse your custom prompt
parser = PromptParser()
scene_data = parser.parse("Your custom scene description")

# Build the scene
builder = UE5SceneBuilder()
builder.build_scene(scene_data)

Expected Output Location

Generated .umap File Location:

YourUE5Project/Content/GeneratedScene.umap

Access in UE5 Editor:

• Path: /Game/GeneratedScene
• Location: Content Browser → Game folder
• File Type: Unreal Map Asset (.umap)

Additional Output Files:

• scene.json - JSON representation of the parsed scene (project root)
• Console logs showing generation progress and material assignments

📁 Project Structure

UEagent2/
├── src/
│   ├── prompt_parser.py      # LLM-powered prompt parsing with GPT-4
│   ├── ue5_scene_builder.py  # Professional greybox scene generation
│   └── __init__.py
├── config/
│   └── scene_schema.json     # JSON schema for scene validation
├── scripts/
│   ├── run_demo.sh          # One-click demo (Unix)
│   └── run_demo.bat         # One-click demo (Windows)
├── tests/
│   └── test_pipeline.py     # Automated testing suite
├── scenes/                   # Base UE5 templates and assets
├── requirements.txt          # Python dependencies
├── PROJECT_STATUS.md         # Development status and features
└── README.md

🚀 Development Roadmap

Current Features ✅

• ✅ GPT-4 powered prompt parsing
• ✅ Professional greybox material system
• ✅ Automated UE5 scene generation
• ✅ Cross-platform compatibility (Windows/macOS)
• ✅ Industry-standard lighting setup
• ✅ Clean object hierarchy and naming

Planned Features 🎯

• 🎯 Material customization API
• 🎯 Advanced room layouts (L-shaped, multi-story)
• 🎯 Prefab object library expansion
• 🎯 Real-time preview mode
• 🎯 Batch processing improvements
• 🎯 Export to other 3D formats

Contributing

We welcome contributions! Areas of focus:

• Additional object types and shapes
• Enhanced material systems
• Performance optimizations
• Documentation improvements

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Support

• Issues: Report bugs and feature requests on GitHub
• Documentation: Check PROJECT_STATUS.md for latest updates
• Community: Join discussions about UE5 automation and greyboxing

---

UE Agent 2 - Transforming natural language into professional UE5 greybox scenes with the power of AI and automation.

⚙️ Configuration

Scene Schema (config/scene_schema.json)

Defines the structure for generated scenes:

• Room specifications with dimensions
• Object positioning and relationships
• Material assignments and properties
• Lighting configuration options

Environment Variables (.env)

OPENAI_API_KEY=your-api-key-here
UE5_PROJECT_PATH=/path/to/your/ue5/project  # Optional

🔧 Troubleshooting

Common Issues

UE5 Python API Not Found

# For UE5.3+ on macOS - Ensure UE5 Python path is in PYTHONPATH
export PYTHONPATH="/Applications/Epic Games/UE_5.3/Engine/Binaries/ThirdParty/Python3/Mac/lib/python3.9/site-packages"

# For UE5.3+ on Windows
set PYTHONPATH="C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\ThirdParty\Python3\Win64\lib\site-packages"

Materials Not Applying

• Verify UE5 BasicShapes plugin is enabled
• Check that material creation isn't blocked by editor settings
• Ensure proper UE5 project structure

Scene Not Saving

• Verify write permissions to UE5 project directory
• Check that UE5 project is properly initialized
• Ensure /Game/ content directory exists

Performance Tips

• Use smaller room dimensions for faster generation
• Limit object count for complex scenes
• Enable UE5 headless mode for batch processing

🎨 Greybox Material System

Professional greybox materials with clear visual differentiation:

Material Type	Color Value	Usage
Walls	0.7 Grey	Room boundaries and vertical surfaces
Floors	0.5 Grey	Ground planes and base surfaces
Ceilings	0.8 Grey	Overhead surfaces
Objects	0.6 Grey	Furniture and props
Doors	0.4 Grey	Entry/exit points
Windows	0.9 Grey	Light-transmitting surfaces

Material Features

• Consistent Naming: All materials use "Greybox_" prefix
• Automatic Application: Materials assigned based on object type
• Industry Standard: Colors follow professional greyboxing conventions
• UE5 Optimized: Uses BasicShapes materials as fallback

💡 Lighting System

Professional 3-point lighting setup for optimal greybox visualization:

• Main Sun: DirectionalLight for primary illumination
• Fill Light: PointLight for shadow softening
• Sky Light: Environmental lighting for realistic ambience
• Intensity Balanced: Prevents harsh shadows while maintaining form definition

⚙️ Configuration

Edit .env file with your settings:

OPENAI_API_KEY=your_openai_api_key_here
UNREAL_ENGINE_PATH=/path/to/UnrealEngine
UE_PROJECT_PATH=/path/to/your/ue5/project
DEFAULT_ROOM_WIDTH=400
DEFAULT_ROOM_LENGTH=500
DEFAULT_ROOM_HEIGHT=250
DEFAULT_TABLE_HEIGHT=75

🔧 Core Components

Prompt Parser

• Uses GPT-4 function calling for structured extraction
• Validates output against JSON schema
• Applies sensible defaults for missing dimensions
• Handles parent-child object relationships

Greybox Scene Builder

• Automated Material System: Creates and applies professional greybox materials
• Optimized Lighting: Sets up industry-standard 3-point lighting
• Professional Naming: Uses "Greybox_" prefixes for clean organization
• UE5 Integration: Spawns geometry with proper positioning and hierarchy
• Clean Architecture: Handles rooms, walls, ceilings, and object placement
• Export Ready: Saves as production-ready .umap files

Supported Elements

• Rooms: bedroom, kitchen, livingroom, bathroom, loft, office
• Objects: bed, table, chair, lamp, sofa, desk, bookshelf, cabinet
• Features: skylight, window, door, fireplace
• Materials: Automatic greybox material assignment by object type

🎮 Usage Examples

Basic Room

from src.prompt_parser import PromptParser

parser = PromptParser()
scene = parser.parse_prompt("Create a bedroom with a bed and nightstand")
parser.save_scene_json(scene, "bedroom.json")

Complex Scene

prompt = "Large living room with a sofa facing a fireplace, coffee table in center, and bookshelf along the wall"
scene = parser.parse_prompt(prompt)

Headless UE5 Build

# Direct UE5 execution
UnrealEditor.exe -run=pythonscript -script="src/ue5_scene_builder.py" "scene.json" -nullrhi

🧪 Testing

Run the test suite:

python -m pytest tests/
# or
python tests/test_pipeline.py

🛠️ Development

Adding New Object Types

1. Update scene_schema.json with new object type
2. Add default dimensions in prompt_parser.py
3. Update function calling schema
4. Test with new prompts

Extending UE5 Integration

• Modify ue5_scene_builder.py for custom materials
• Add more sophisticated geometry
• Implement proper lighting setups

📝 Output

The pipeline generates:

• scene.json - Structured scene graph
• /Game/Maps/BlockOut.umap - UE5 level file
• Console logs showing build progress

🔍 Troubleshooting

Common Issues:

• Missing OpenAI API key → Check .env file
• UE5 path not found → Verify UNREAL_ENGINE_PATH
• JSON validation errors → Check prompt complexity
• UE5 Python import errors → Ensure UE5 Python environment

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Add tests for new functionality
4. Submit a pull request

📚 References

• Unreal Engine Python API
• OpenAI Function Calling
• JSON Schema