metadata
library_name: transformers
pipeline_tag: text-to-image
InstanceControl: Controllable Complex Image Generation without Instance Labeling
This repository contains the Stage 1 model (Sa2va-Instance-4B) for InstanceControl, a multi-instance controllable generation method that eliminates the need for manual instance labeling.
- Paper: InstanceControl: Controllable Complex Image Generation without Instance Labeling
- Project Page: InstanceControl Project Page
- Repository: GitHub - InstanceControl
Method Overview
InstanceControl leverages a Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions (such as Canny edge, depth, or HED maps). It automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. An adaptive mask refinement strategy dynamically refines these instance masks during the generation process to achieve superior fidelity and precise instance-level control.
Citation
If you find this work useful, please consider citing:
@article{instancecontrol,
title = {InstanceControl: Controllable Complex Image Generation without Instance Labeling},
author = {Xiaoyu Liu and Huan Wang and Fan Li and Zhixin Wang and Jiaqi Xu and Ming Liu and Wangmeng Zuo},
journal = {arXiv preprint arXiv:2606.31924},
year = {2026}
}