LatentDance: Towards Realistic and Dynamic Character Animation via Identity-Aware Motion Representation

Paper

This repository is the official PyTorch implementation of our paper, LatentDance: Towards Realistic and Dynamic Character Animation via Identity-Aware Motion Representation.

LatentDance animates a reference character from a pose sequence while preserving identity details and producing realistic, dynamic motion.

Yixin Yang, Yeying Jin, Jiawei Zhang, Long Sun, Xu Cheng, Jinshan Pan

🔥 News

• [2026-08-04] Released the paper, inference code, gradio demo and pretrained LatentDance checkpoints.

Requirements

• Linux and Python 3.10+
• NVIDIA GPU with CUDA support, ~80GB VRAM for Single-GPU inference.
• PyTorch 2.8.0 and torchvision 0.23.0
• FFmpeg with the libx264 encoder for H.264 pose-video output. The Python dependencies also install an imageio-ffmpeg fallback.
• Four GPUs are recommended for the released 14B configuration. Single-GPU execution is possible by setting NPROC_PER_NODE=1, subject to available VRAM.

:briefcase: Installation

git clone https://github.com/yyang181/LatentDance.git
cd LatentDance

conda create -n latentdance python=3.10 -y
conda activate latentdance

pip install torch==2.8.0 torchvision==0.23.0 \
  --index-url https://download.pytorch.org/whl/cu128
pip install -e ".[multi-gpu]"

# Recommended for faster attention on NVIDIA GPUs.
FLASH_ATTENTION_FORCE_BUILD=TRUE pip install flash_attn --no-build-isolation

# Optional when a system ffmpeg is not already available. This provides libx264.
conda install -c conda-forge ffmpeg

Verify that the selected FFmpeg build includes H.264 encoding:

ffmpeg -hide_banner -encoders | grep libx264

If the command prints a libx264 encoder, pose extraction will write H.264 (yuv420p) MP4 files. When no system ffmpeg is found, the imageio[ffmpeg] dependency installed above supplies a bundled encoder automatically.

:gift: Checkpoints

Download the Wan2.2-I2V-A14B base model and the two LatentDance denoisers:

mkdir -p models/Wan-AI models/LatentDance

HF_HUB_ENABLE_HF_TRANSFER=1 hf download Wan-AI/Wan2.2-I2V-A14B \
  --local-dir models/Wan-AI/Wan2.2-I2V-A14B

HF_HUB_ENABLE_HF_TRANSFER=1 hf download yyang181/LatentDance \
  --local-dir models/LatentDance

The resulting layout should be:

models/
├── LatentDance/
│   ├── latentdance_high_noise.safetensors
│   └── latentdance_low_noise.safetensors
└── Wan-AI/
    └── Wan2.2-I2V-A14B/
        └── ...

Pose and Keypoint Preprocessing

LatentDance inference consumes a rendered pose video and the corresponding DWPose coordinates. This repository includes a minimal preprocessing CLI that creates both from ordinary driving videos.

Install the pose extraction environment:

pip install -e ".[pose]"

The pose extra installs ONNX Runtime with CUDA support. For CPU-only preprocessing, replace it with the CPU build:

pip uninstall -y onnxruntime-gpu
pip install onnxruntime

Place driving videos under video/, then run preprocessing:

data/evaldata/
├── input_image/
│   └── 000.png
└── video/
    └── 000.mp4

CUDA_VISIBLE_DEVICES=0 python scripts/extract_pose.py \
  --input_dir data/evaldata \
  --body_score_threshold 0.3

On the first run, the script downloads yolox_l.onnx and dw-ll_ucoco_384.onnx from yzd-v/DWPose. In an offline environment, download them in advance and pass --detector_model /path/to/yolox_l.onnx --pose_model /path/to/dw-ll_ucoco_384.onnx. Add --cpu to explicitly disable CUDA or --overwrite to regenerate existing results.

For each video/<name>.mp4, preprocessing writes:

data/evaldata/
├── pose/<name>.mp4             # H.264 DWPose rendering (yuv420p), with input FPS and resolution
└── pose_keypoints/<name>.npz   # bodies_candidate: float32 [frames, 128, 3]

Each keypoint stores normalized (x, y, confidence). The 128 points are ordered as 18 body joints, 68 face landmarks, 21 right-hand joints, and 21 left-hand joints, exactly matching the released inference loader. Input videos may be nested under video/; the same relative structure is preserved in both output directories.

Evaluation Data

The released inference pipeline matches reference images, pose videos, keypoints, and captions by filename stem. Prepare the evaluation data as follows:

data/evaldata/
├── input_image/
│   └── 000.png
├── video/
│   └── 000.mp4
├── pose/
│   └── 000.mp4
├── pose_keypoints/
│   └── 000.npz
└── pllava_caption/
    └── caption.csv

The caption CSV must contain path and text columns. For example:

path,text
video/000.mp4,"A person is dancing."

:zap: Inference

Run the inference pipeline with the public checkpoints:

CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 bash scripts/batch_test.sh

For a quick smoke test on the first sample:

EVAL_LIMIT=1 CUDA_VISIBLE_DEVICES=0,1,2,3 NPROC_PER_NODE=4 bash scripts/batch_test.sh

Generated videos are saved under output/ by default. Set OUTPUT_DIR=/path/to/output to override the output directory, EVALUATION_PATH=/path/to/data to use another evaluation set, or CHUNK_SIZE=161 to change the temporal chunk size.

Gradio Demo

Launch an interactive UI for one reference image and one ordinary driving video. The demo separates DWPose extraction from final video generation and uses the same inference implementation as the batch evaluator.

pip install -e ".[demo,pose,multi-gpu]"
python demo.py

Open http://127.0.0.1:7860 in your browser. The demo uses two stages:

The English / 中文 button switches the visible usage instructions without changing any inputs or intermediate results.

1. Upload a reference image and driving video, select one or more GPUs, and click Extract Pose. This produces the pose visualization and an .npz keypoints file. Final generation uses the reference image, the rendered pose control video, and the selected .npz keypoints.
2. Click Generate Video to run LatentDance. The generation stage uses the keypoints from the extraction stage by default. You may instead upload another compatible .npz file in the keypoints input before generating.

If extraction has not been run for the current inputs, Generate Video automatically prepares the required pose video first; an uploaded .npz still takes priority for keypoints.

Selecting one GPU runs normal single-process inference; selecting multiple GPUs launches one process per selected GPU with USP sequence parallelism. The default checkpoint paths are models/LatentDance/latentdance_high_noise.safetensors and models/LatentDance/latentdance_low_noise.safetensors; they can be changed in the UI. A CUDA-capable GPU, the pose extraction dependencies, xFuser for multi-GPU inference, and the Wan2.2-I2V-A14B base model are required.

To Do

• Release testing code
• Release pre-trained models
• Release Gradio demo
• [] Release training code

Citation

If this work is useful for your research, please cite the LatentDance paper. The BibTeX entry will be added when the paper page is public.

License

This repository is released under the Apache License 2.0. Please also comply with the licenses of Wan2.2, DiffSynth-Studio, xDiT/xFuser, and other third-party dependencies.

Acknowledgement

This codebase builds on DiffSynth-Studio and Wan2.2. We thank the authors for their excellent open-source work.

Contact

This repository is maintained by Yixin Yang (@yyang181).