ComfyUI-Kaola-ACE-Step

ComfyUI custom nodes for ACE-Step 1.5 - Commercial-grade music generation.

Features

🎵 Text to Music - Generate music from text
🎭 Cover Generation - Style transfer
🎨 Audio Repaint - Local audio editing
💡 Simple Mode - Natural language to music
📝 Format Sample - Enhance user input
🔍 Understand Audio - Analyze audio codes
🎤 Audio Transcription - Transcribe lyrics from audio
📋 Audio Captioning - Generate detailed music descriptions
🎸 Extract/Lego/Complete - Track extraction and manipulation (Base model)
🧹 Clear VRAM - Free GPU memory after operations

Quick Start

1. Install ACE-Step

git clone https://github.com/ACE-Step/ACE-Step-1.5.git
cd ACE-Step-1.5
pip install -e .

2. Install ComfyUI Nodes

Standard Installation:

cd ComfyUI/custom_nodes
git clone https://github.com/kana112233/ComfyUI-kaola-ace-step.git
cd ComfyUI-kaola-ace-step
pip install -r requirements.txt

For ComfyUI Portable:

# Navigate to your ComfyUI portable folder
cd ComfyUI_windows_portable/ComfyUI/custom_nodes
git clone https://github.com/kana112233/ComfyUI-kaola-ace-step.git

# Use the embedded Python to install requirements
cd ComfyUI-kaola-ace-step
../../python_embeded/python.exe -m pip install -r requirements.txt

# You must also install ACE-Step in the embedded Python:
cd ../../../
git clone https://github.com/ACE-Step/ACE-Step-1.5.git
python_embeded/python.exe -m pip install -e ACE-Step-1.5

3. Download Models

# Using huggingface-cli
huggingface-cli download ACE-Step/Ace-Step1.5 --local-dir ComfyUI/models/Ace-Step1.5

# Or using hfd (faster)
pip install hfd
hfd ACE-Step/Ace-Step1.5 --local-dir ComfyUI/models/Ace-Step1.5

4. Restart ComfyUI

Node Settings

Parameter	Value	Description
`checkpoint_dir`	Leave empty	Uses `ComfyUI/models/Ace-Step1.5/` by default
`config_path`	`acestep-v15-turbo`	Recommended for speed
`lm_model_path`	`acestep-5Hz-lm-1.7B`	Recommended language model
`device`	`auto`	Auto-detects device

Model Directory Structure

The checkpoint_dir dropdown scans all folders directly under ComfyUI/models/. You can choose any folder from the root models directory as your ACE-Step base folder.

Example layout:

ComfyUI/models/
├── Ace-Step1.5/            # Select "Ace-Step1.5" in the node
│   ├── acestep-v15-turbo/
│   ├── vae/
│   └── ...
└── My-Custom-Folder/       # Select "My-Custom-Folder" in the node
    ├── acestep-v15-turbo/
    ├── vae/
    └── ...

Workflow Examples

See examples/ directory for ready-to-use workflows.

Nodes

Node	Description
ACE_STEP_TextToMusic	Generate music from text
ACE_STEP_Cover	Style transfer
ACE_STEP_Repaint	Local audio editing
ACE_STEP_SimpleMode	Natural language generation
ACE_STEP_FormatSample	Format and enhance input
ACE_STEP_Understand	Analyze audio codes
ACE_STEP_Transcriber	Transcribe lyrics from audio (50+ languages)
ACE_STEP_Captioner	Generate detailed music descriptions
ACE_STEP_Extract	Extract instrument/vocal tracks (Base model)
ACE_STEP_Lego	Add/modify instrument tracks (Base model)
ACE_STEP_Complete	Complete missing tracks (Base model)
ACE_STEP_ClearVRAM	Free GPU memory

📖 Detailed usage guide: See USAGE.md for comprehensive documentation.

ACE_STEP_TRANSCRIBER

Audio transcription node powered by ACE-Step Transcriber (Qwen2.5-Omni-7B).

Features

🌍 50+ Languages - Chinese, English, Japanese, Korean, etc.
🎵 Lyrics Transcription - Specialized for singing voice
🏷️ Structure Tags - Auto identifies [Verse], [Chorus], [Bridge], etc.
⏱️ Timestamps - Via prompt instruction
✂️ Chunked Processing - Handle long audio automatically

Installation

Download the model to your models folder:

# Using huggingface-cli
huggingface-cli download ACE-Step/acestep-transcriber --local-dir ComfyUI/models/acestep-transcriber

# Or place inside Ace-Step1.5 folder
huggingface-cli download ACE-Step/acestep-transcriber --local-dir ComfyUI/models/Ace-Step1.5/acestep-transcriber

Parameters

Parameter	Default	Description
`language`	auto	Target language (auto/zh/en/ja/ko/etc.)
`chunk_length_s`	30	Audio chunk length in seconds
`max_new_tokens`	4096	Max output length
`temperature`	0.2	Sampling temperature
`top_p`	0.95	Nucleus sampling threshold
`repetition_penalty`	1.1	Penalty for repeating tokens
`num_beams`	1	Beam search (higher = slower but better)
`seed`	0	Random seed (0 = random)
`custom_prompt`	""	Override default prompt

ACE_STEP_Captioner

Audio captioning node powered by ACE-Step Captioner (Qwen2.5-Omni-7B).

Features

🎼 Musical Style Analysis - Identifies genres, sub-genres, and stylistic influences
🎸 Instrument Recognition - Detects and describes 1000+ instrument types
🎭 Structure & Progression - Analyzes musical arrangement (intro, verse, chorus, bridge, etc.)
🔊 Timbre Description - Captures tonal qualities, textures, and sonic characteristics

Performance

Accuracy surpasses Gemini Pro 2.5 in music description tasks.

Installation

Download the model to your models folder:

# Using huggingface-cli
huggingface-cli download ACE-Step/acestep-captioner --local-dir ComfyUI/models/Ace-Step1.5/acestep-captioner

# Or using hfd (faster)
pip install hfd
hfd ACE-Step/acestep-captioner --local-dir ComfyUI/models/Ace-Step1.5/acestep-captioner

Parameters

Parameter	Default	Description
`model_id`	Ace-Step1.5/acestep-captioner	Model path (local or HuggingFace ID)
`device`	auto	Device (auto/cuda/cpu/mps)
`dtype`	auto	Precision (auto/float16/float32)
`custom_prompt`	Task Describe this audio in detail	Prompt for captioning
`max_new_tokens`	1024	Max output length
`temperature`	0.3	Sampling temperature
`top_p`	0.9	Nucleus sampling threshold
`top_k`	50	Top-K sampling
`repetition_penalty`	1.1	Penalty for repeating tokens
`seed`	0	Random seed (0 = random)
`chunk_length_s`	30	Audio chunk length for long audio

Outputs

Output	Description
`caption`	Concise one-sentence summary
`style_tags`	Comma-separated style/instrument tags
`full_description`	Complete detailed description

Requirements

Python: 3.10+
GPU: 6GB+ VRAM recommended
Disk: ~8GB for models

6. Using LoRA Models (e.g., Chinese New Year)

Use the ACE-Step LoRA Loader node to apply LoRA adaptations.

Place your LoRA model folder (containing adapter_config.json) in ComfyUI/models/loras/ or ComfyUI/models/Ace-Step1.5/loras/.
Add the ACE-Step LoRA Loader node.
Select your LoRA from the dropdown.
Connect the lora_info output to any ACE-Step generation node (TextToMusic, etc.).
Adjust strength as needed (default 1.0).

Troubleshooting

"Model path not found"

Ensure models are in ComfyUI/models/Ace-Step1.5/ with the correct subdirectory structure (see above).

Audio Transcription Technical Roadmap

graph TD
    Start[🚀 Start: Select Audio Transcription Solution] --> Domain{Core Use Case?}

    %% Branch 1: Music & Lyrics
    Domain -- Music / Lyrics / Structure --> Music[Music Domain]
    Music --> Hardware1{VRAM > 30GB?}
    Hardware1 -- Yes --> ACE[<b>ACE-Step Transcriber</b><br>Base: Qwen2.5-Omni-7B]
    ACE --> ACEDep[<u>Features</u>:<br>✅ 50+ Languages Support<br>✅ Lyrics + Structure Tags<br>✅ Timestamps via Prompt]
    Hardware1 -- No --> Heart[<b>HeartTranscriptor-oss</b><br>0.8B Params]

    %% Branch 2: Speech & Meetings
    Domain -- Speech / Meetings / Translation --> Speech[Speech Domain]
    Speech --> Duration{Duration & Coherence?}

    %% VibeVoice Path
    Duration -- Long Meeting --> Vibe[<b>VibeVoice-ASR</b><br>64K Context]

    %% Whisper Path
    Duration -- General / Short Audio --> Whisp[<b>Whisper-large-v3</b><br>Mature Ecosystem]
    Whisp --> Speed{Speed Priority?}
    Speed -- Fast --> WhispFast[torch.compile]
    Speed -- Long Audio --> WhispLong[Pipeline Chunking]

    %% Styles
    classDef model fill:#e1f5fe,stroke:#01579b,stroke-width:2px;
    classDef dep fill:#fff3e0,stroke:#ff6f00,stroke-width:1px,stroke-dasharray: 5 5;
    class ACE,Heart,Vibe,Whisp,WhispFast,WhispLong model;
    class ACEDep dep;

📋 Technical Dependency Matrix

Feature	ACE-Step Transcriber	VibeVoice-ASR	Whisper-large-v3	HeartTranscriptor-oss
🏗️ Core Arch	Qwen2.5-Omni (Multimodal LLM)	VibeVoice (Unified ASR)	Transformer (Encoder-Decoder)	Whisper Variant
📦 Library Deps	Transformers (Preview), qwen-omni-utils, decord	vLLM (Recommended), Transformers	Transformers, flash-attn (Optional), accelerate	heartlib (GitHub), safetensors
🧠 VRAM/Hardware	Very High (~30GB+), Flash Attn 2 Recommended	High (64K Context, Single-Pass)	Medium (~24GB, Consumer GPU OK)	Low (0.8B, Edge devices)
✂️ Input Logic	Prompt Driven, requires process_mm_info	Long Context (60min Single-Pass), Hotwords	Chunking (30s limit), chunk_length_s=30	Standard (heartlib loader)
🚫 Constraints	Version Lock: Qwen-Preview branch required	Memory Wall: Cannot chunk, must load 1hr at once	torch.compile incompatible with Chunking & Flash Attn 2	Eco-Isolation: Independent codebase

License

MIT License - see LICENSE for details.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
acestep_repo @ 45d9783		acestep_repo @ 45d9783
docs		docs
examples		examples
.gitignore		.gitignore
.gitmodules		.gitmodules
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
MODEL_SETUP.md		MODEL_SETUP.md
README.md		README.md
USAGE.md		USAGE.md
__init__.py		__init__.py
acestep_wrapper.py		acestep_wrapper.py
nodes.py		nodes.py
nodes_base_features.py		nodes_base_features.py
nodes_captioner.py		nodes_captioner.py
nodes_transcriber.py		nodes_transcriber.py
nodes_vram.py		nodes_vram.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt

Navigation Menu

Repository navigation

About

Repository files navigation

ComfyUI-Kaola-ACE-Step

Features

Quick Start

1. Install ACE-Step

2. Install ComfyUI Nodes

3. Download Models

4. Restart ComfyUI

Node Settings

Model Directory Structure

Workflow Examples

Nodes

ACE_STEP_TRANSCRIBER

Features

Installation

Parameters

ACE_STEP_Captioner

Features

Performance

Installation

Parameters

Outputs

Requirements

6. Using LoRA Models (e.g., Chinese New Year)

Troubleshooting

"Model path not found"

Audio Transcription Technical Roadmap

📋 Technical Dependency Matrix

License

Links

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

ComfyUI-Kaola-ACE-Step

Features

Quick Start

1. Install ACE-Step

2. Install ComfyUI Nodes

3. Download Models

4. Restart ComfyUI

Node Settings

Model Directory Structure

Workflow Examples

Nodes

ACE_STEP_TRANSCRIBER

Features

Installation

Parameters

ACE_STEP_Captioner

Features

Performance

Installation

Parameters

Outputs

Requirements

6. Using LoRA Models (e.g., Chinese New Year)

Troubleshooting

"Model path not found"

Audio Transcription Technical Roadmap

📋 Technical Dependency Matrix

License

Links

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages