LatentSync

Name: LatentSync
Author: jiafuzeng

We present LatentSync, an end-to-end lip-sync method based on audio-conditioned latent diffusion models without any intermediate motion representation, diverging from previous diffusion-based lip-sync methods based on pixel-space diffusion or two-stage generation. Our framework can leverage the powerful capabilities of Stable Diffusion to directly model complex audio-visual correlations.

VRAM Requirements

Direct Answer: The ComfyUI node LatentSync requires a minimum base VRAM of 4096MB and is optimized for GPUs with at least 8GB of VRAM. Low VRAM mode is not supported for this node.

Very High (4-8GB)

Base VRAM:: 4096MB (4.0GB)
Recommended GPU:: 8GB+ VRAM
Low VRAM Mode:: ✗ Not supported
Estimation Confidence:: MEDIUM

🧮 Interactive VRAM Compatibility Estimator

1. Your GPU VRAM:

2. Base Model Weight:

3. Target Resolution:

4. ComfyUI Optimization launch argument:

Estimated Total VRAM: 3.00 GBTarget: 8 GB

✅ Comfortable Fit

Your GPU has plenty of headroom. You can run this node safely with your active configurations!

Deploy on High-Performance GPUs

Need more VRAM to run ComfyUI with this node? Rent low-cost, high-performance GPUs on Vast.ai instantly.

🚀 Deploy on Vast.ai

Deploy on Cloud GPUs

Need more VRAM to run ComfyUI with this node? Rent low-cost, high-performance cloud GPUs on RunPod instantly.

🚀 Deploy on RunPod

Python Dependencies

Direct Answer: Running LatentSync requires installing the following Python package dependencies: accelerate==0.26.1, av==11.0.0, decord==0.6.0, diffusers==0.32.2, einops==0.7.0, face-alignment==1.4.1, ffmpeg-python==0.2.0, gradio==5.12.0, huggingface-hub==0.25.2, imageio==2.27.0, librosa==0.10.1, lpips==0.1.4, mediapipe==0.10.11, ninja==1.11.1.1, numpy==1.26.4, omegaconf==2.3.0, opencv-python==4.9.0.80, pandas==2.0.3, python_speech_features==0.6, safetensors==0.4.2, scenedetect==0.6.1, torch==2.4.1, torch-fidelity==0.3.0, torchmetrics==1.3.1, torchvision==0.19.1, transformers==4.48.0. Ensure your ComfyUI environment has these packages active before launching.

requirements.txt

accelerate==0.26.1
av==11.0.0
decord==0.6.0
diffusers==0.32.2
einops==0.7.0
face-alignment==1.4.1
ffmpeg-python==0.2.0
gradio==5.12.0
huggingface-hub==0.25.2
imageio==2.27.0
librosa==0.10.1
lpips==0.1.4
mediapipe==0.10.11
ninja==1.11.1.1
numpy==1.26.4
omegaconf==2.3.0
opencv-python==4.9.0.80
pandas==2.0.3
python_speech_features==0.6
safetensors==0.4.2
scenedetect==0.6.1
torch==2.4.1
torch-fidelity==0.3.0
torchmetrics==1.3.1
torchvision==0.19.1
transformers==4.48.0

🛠️ Interactive Setup & Dependency Resolver

Operating System:

Environment Type:

Run this terminal command in your ComfyUI root folder:

# Loading command...