Add model card for Sa2va-Instance-4B (Stage 1 of InstanceControl)

README.md

@@ -0,0 +1,63 @@
+---
+license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+---
+
+# Sa2va-Instance-4B (Stage 1 of InstanceControl)
+
+This repository contains the **Sa2va-Instance-4B** model checkpoint, which serves as the Stage 1 (instance parsing/mask prediction) model of **InstanceControl**, presented in the paper [InstanceControl: Controllable Complex Image Generation without Instance Labeling](https://huggingface.co/papers/2606.31924).
+
+* **Project Page:** [InstanceControl Homepage](https://instancecontrol.github.io/InstanceControl/)
+* **Repository:** [GitHub - liuxiaoyu1104/InstanceControl](https://github.com/liuxiaoyu1104/InstanceControl)
+* **Dataset (MIG-Train):** [MIG_train](https://huggingface.co/datasets/xiaoyu1104/MIG_train)
+* **Evaluation Benchmark:** [MIG-Eval](https://huggingface.co/datasets/xiaoyu1104/MIG-Eval)
+
+---
+
+## Model Description
+
+InstanceControl is a multi-instance controllable generation method that eliminates the need for manual instance labeling. 
+
+**Sa2va-Instance-4B** is a Vision-Language Model (VLM) designed to establish instance-level correspondences between text prompts and visual conditions (such as Canny edge maps, depth maps, or HED boundary maps). It automatically parses instance descriptions from the text prompts and predicts instance masks based on the visual conditions. These predicted masks are then dynamically refined during the Stage 2 generation process using a modified FLUX ControlNet.
+
+---
+
+## Installation
+
+To set up the environment for running Sa2va-Instance-4B, follow these steps:
+
+```bash
+conda create -n instancecontrol python=3.10
+conda activate instancecontrol
+
+pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
+pip install -r requirements.txt
+```
+
+## Quick Start (Stage 1 Inference)
+
+To predict instance masks from your visual conditions and prompt JSON files, place the model checkpoint under `pretrain_model/InstanceControl_canny/Sa2va-Instance-4B` and run:
+
+```bash
+python stage1_Sa2VA/projects/llava_sam2/evaluation/gcg_eval_our_folders.py \
+  --model_path ./pretrain_model/InstanceControl_canny/Sa2va-Instance-4B \
+  --image_dir ./example/canny \
+  --json_dir ./example/json \
+  --save_dir ./results/json_pred_canny
+```
+
+For more details on training, dataset formatting, and generating final images using the Stage 2 FLUX ControlNet, please refer to the [GitHub Repository](https://github.com/liuxiaoyu1104/InstanceControl).
+
+## Citation
+
+If you find this model or the project useful, please cite:
+
+```bibtex
+@article{instancecontrol,
+  title   = {InstanceControl: Controllable Complex Image Generation without Instance Labeling},
+  author  = {Xiaoyu Liu and Huan Wang and Fan Li and Zhixin Wang and Jiaqi Xu and Ming Liu and Wangmeng Zuo},
+  journal = {arXiv preprint arXiv:2606.31924},
+  year    = {2026}
+}
+```