InstructAV2AV: Instruction-Guided Audio-Video Joint Editing

Haojie Zheng^{1, 2}, Yixin Yang², Siqi Yang², Shuchen Weng^{1, 2*}, Boxin Shi^2*
*Corresponding authors
¹BAAI, ²Peking University

Overview

InstructAV2AV is an instruction-guided framework for joint audio-video editing. Given a source video with audio and a natural-language instruction, the model edits the visual and audio content together while preserving the unedited properties of the source as much as possible.

This repository contains six task-specific InstructAV2AV checkpoints. They share the same architecture and inference pipeline, but are fine-tuned for different editing objectives, including general editing, content insertion and removal, visual identity preservation, voice preservation, and joint identity-and-voice preservation.

The inference and training code is available in the InstructAV2AV GitHub repository. The corresponding training dataset is available as InsAVE-80K.

Video Demo

For the best experience, please enable audio.

Checkpoints

Checkpoint	Editing type	Description	Example instruction
general.safetensors	General editing	Edits appearance, scenes, actions, speech, and sound.	Make the horse dark brown with a white saddle.
insertion.safetensors	Content insertion	Adds an object or other content to the source video.	Add a dark vintage sedan driving from the right to the left.
removal.safetensors	Content removal	Removes an object and its associated audiovisual content.	Remove the chipmunk standing on the stone surface among the peanuts.
clone_id.safetensors	Identity cloning	Preserves a person's visual identity while editing other visual or audio attributes.	Keep the person's appearance, change the timbre to a man, and change the spoken words to <S>I understand, but I think we need to consider.<E>.
clone_voice.safetensors	Voice cloning	Preserves the speaker's timbre while editing the video or spoken content.	Keep the timbre, change the person's appearance, and change the spoken words to <S>I came here to tell you that you should go.<E>.
clone_id_voice.safetensors	Identity and voice cloning	Preserves both the person's visual identity and the speaker's timbre.	Keep the person's identity and voice, and change the spoken words to <S>This is more than just art, it's a statement.<E>.

Select the checkpoint that most closely matches the requested edit and pass its path through --finetune-path.

Installation

The checkpoints must be used with the InstructAV2AV codebase.

git clone https://github.com/suimuc/InstructAV2AV.git
cd InstructAV2AV

conda create -n instructav2av python=3.10 -y
conda activate instructav2av

pip install torch==2.6.0 torchvision torchaudio
pip install -r requirements.txt
pip install flash_attn --no-build-isolation
pip install -e .

The provided installation instructions target CUDA 12.1.

Download

Download all six InstructAV2AV checkpoints:

huggingface-cli download suimu/InstructAV2AV \
  --local-dir ckpts/InstructAV2AV

The inference pipeline also requires the Wan2.2 text encoder and video VAE, together with the MMAudio audio VAE and vocoder:

# Text encoder and video VAE
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B \
  --include \
    "google/*" \
    "models_t5_umt5-xxl-enc-bf16.pth" \
    "Wan2.2_VAE.pth" \
  --local-dir ckpts/Wan2.2-TI2V-5B

# Audio VAE and vocoder
huggingface-cli download hkchengrex/MMAudio \
  --include \
    "ext_weights/best_netG.pt" \
    "ext_weights/v1-16.pth" \
  --local-dir ckpts/MMAudio

The resulting checkpoint layout should include:

ckpts/
├── InstructAV2AV/
│   ├── clone_id.safetensors
│   ├── clone_id_voice.safetensors
│   ├── clone_voice.safetensors
│   ├── general.safetensors
│   ├── insertion.safetensors
│   └── removal.safetensors
├── MMAudio/
│   └── ext_weights/
│       ├── best_netG.pt
│       └── v1-16.pth
└── Wan2.2-TI2V-5B/
    ├── google/
    ├── models_t5_umt5-xxl-enc-bf16.pth
    └── Wan2.2_VAE.pth

Inference

The unified command-line entry point is scripts/edit.py. The source video must contain an audio track unless a separate audio file is supplied with --source-audio.

Single-video editing

python scripts/edit.py \
  --source-video assets/input.mp4 \
  --instruction "Keep the person's identity and change the spoken words to <S>This is more than just art, it's a statement.<E>." \
  --finetune-path ckpts/InstructAV2AV/clone_id_voice.safetensors \
  --output outputs/edited.mp4

To use a separate audio track:

python scripts/edit.py \
  --source-video assets/input.mp4 \
  --source-audio assets/input.wav \
  --instruction "Make the horse dark brown with a white saddle." \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output outputs/edited.mp4

Prompt enhancement

When specifying new spoken content, place the target utterance between <S> and <E> markers:

Change the spoken words to <S>This is the new sentence.<E>.

For better editing quality and text-to-video alignment, append a detailed description of the desired final video immediately after the editing command. The enhanced prompt can describe the subject's appearance, clothing, actions, expressions, scene, and target speech. For example:

python scripts/edit.py \
  --source-video assets/input1.mp4 \
  --instruction "Change the man into a young woman with brown hair, wearing a gray blazer over a light pink top and a necklace with a heart-shaped pendant, and saying, <S>I really think we should give it another chance.<E>. A young woman with long, wavy brown hair and fair skin is engaged in a conversation. She is wearing a gray blazer over a light pink top and has a necklace with a heart-shaped pendant. Her facial expressions change throughout the sequence, showing a range of emotions that suggest she is either explaining something earnestly or reacting to a conversation, and says <S>I really think we should give it another chance.<E>. The setting appears to be indoors, with a dimly lit, blurred background that suggests a social environment, possibly a bar or restaurant. The focus remains on the woman's face, capturing her reactions and engagement in the dialogue." \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output outputs/edited.mp4

Gradio demo

After downloading all six checkpoints and the dependency weights, launch the interactive demo with:

python scripts/demo.py --share

Default Inference Configuration

The released configuration uses 121 frames at 704 × 1280 resolution, 16 kHz audio, 50 sampling steps, and bfloat16 inference. CUDA is required. See ovi/configs/inference/inference_av_edit.yaml in the code repository for all options, including guidance scales, seed, sequence parallelism, and CPU offloading.

InsAVE-80K Dataset

The training dataset is available at suimu/InsAVE-80K.

Training

The unified training entry point is scripts/train.py, with configuration in ovi/configs/train/train_av_edit.yaml. The trainer accepts CSV, JSON, or JSONL manifests with the following canonical fields:

Field	Description
source_video	Original video
source_audio	Original audio
target_video	Edited video
target_audio	Edited audio
instruction	Editing instruction

Before training, set the checkpoint directory, initialization checkpoint, and dataset manifest in the training configuration:

ckpt_dir: ./ckpts
finetune_path: ./ckpts/InstructAV2AV/general.safetensors

dataset:
  metadata_path: ./data/InsAVE-80K/path/to/manifest.csv

The manifest and initialization checkpoint can also be overridden from the command line:

accelerate launch \
  --config_file ovi/configs/train/accelerate_config.yaml \
  scripts/train.py \
  --config-file ovi/configs/train/train_av_edit.yaml \
  --data-manifest data/InsAVE-80K/path/to/manifest.csv \
  --finetune-path ckpts/InstructAV2AV/general.safetensors \
  --output-dir outputs/train_av_edit

The provided Accelerate configuration uses DeepSpeed ZeRO-2 and is configured for eight processes. Change num_processes in ovi/configs/train/accelerate_config.yaml to match the available GPUs.

The training objective is selected from the modality flags:

has_video	has_audio	Training objective
true	true	Joint audio-video editing
true	false	Video-only editing
false	true	Audio-only editing

Related Resources

• Source code
• Project page
• InsAVE-80K dataset
• Wan2.2
• MMAudio
• Ovi

Citation

If you find InstructAV2AV or InsAVE-80K useful in your research, please cite:

@article{instructav2av2026,
  title={InstructAV2AV: Instruction-Guided Audio-Video Joint Editing},
  author={Zheng, Haojie and Yang, Yixin and Yang, Siqi and Weng, Shuchen and Shi, Boxin},
  journal={arXiv preprint arXiv:2605.18467},
  year={2026}
}

License

The project is released under the Apache License 2.0. Users must also comply with the licenses and terms of the underlying pretrained models and any input media or datasets they use.