HeteroTissueDiffuse

Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology

NeurIPS 2025

Saghir Alfasly · Wataru Uegami · MD Enamul Hoq · Ghazal Alabtah · H.R. Tizhoosh

KIMIA Lab, Department of AI & Informatics, Mayo Clinic, Rochester, MN, USA

---

Model Description

HeteroTissueDiffuse is a latent diffusion model (LDM) that synthesizes heterogeneous histopathology images by conditioning on both a binary semantic map and raw tissue crop exemplars. Unlike text- or embedding-guided approaches, it injects actual tissue appearance directly into the diffusion process, preserving staining characteristics, nuclear morphology, and cellular texture.

The model addresses a fundamental limitation of prior generative methods in histopathology: the tendency to produce homogeneous (single-tissue-type) images. By providing spatially-registered visual prompts for each tissue class, the model generates realistic heterogeneous slides that accurately reflect real-world tissue organization.

Architecture

• Base: CompVis Latent Diffusion Model with VQ-regularized autoencoder
• First stage: VQModelInterface (3-channel latent, 8192 codebook)
• Conditioning encoder: SpatialRescaler with in_channels=8 (replaces ADE20K default of 182)
• U-Net: 128 base channels, attention at resolutions 32/16/8
• Image size: 256×256 pixels
• Sampling: DDIM, 200 steps, η=1

8-Channel Conditioning Tensor

Channel 0:    normal onehot mask         (1 where segmentation == 0)
Channels 1–3: normal tissue crop RGB     (float32, normalized to [-1,1])
Channel 4:    tumor onehot mask          (1 where segmentation == 1)
Channels 5–7: tumor tissue crop RGB      (float32, normalized to [-1,1])

The tissue crops are small patches (typically 30–60px) extracted from a reference slide and pasted spatially within the corresponding mask region. This lets users control staining appearance at inference time without any fine-tuning.

---

Available Checkpoints

File	Dataset	Description
camelyon16/epoch=000064.ckpt	Camelyon16	Binary tumor/normal masks, 256×256, 64 epochs
panda/last.ckpt	PANDA	Gleason tissue regions, 256×256
tcga/last.ckpt	TCGA (self-supervised)	100 pseudo-phenotype clusters, 256×256, 232 epochs

---

Quick Start

1. Clone the inference code

git clone https://github.com/CompVis/stable-diffusion.git
cd stable-diffusion

# Apply the 2 required patches (see GitHub README for details)
# Then copy our inference script
wget https://raw.githubusercontent.com/Saghir/HeteroTissueDiffuse/main/inference_heteroTissueDiffuse_camelyon.py

2. Download the checkpoint

pip install huggingface_hub

python - <<'EOF'
from huggingface_hub import hf_hub_download
# Camelyon16
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="camelyon16/epoch=000064.ckpt", local_dir="inference/")
# PANDA
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="panda/last.ckpt", local_dir="inference/")
# TCGA
hf_hub_download(repo_id="Saghir/HeteroTissueDiffuse",
                filename="tcga/last.ckpt", local_dir="inference/")
EOF

Or via CLI (download one dataset at a time):

# Camelyon16
huggingface-cli download Saghir/HeteroTissueDiffuse \
    camelyon16/epoch=000064.ckpt --local-dir inference/

# PANDA
huggingface-cli download Saghir/HeteroTissueDiffuse \
    panda/last.ckpt --local-dir inference/

# TCGA
huggingface-cli download Saghir/HeteroTissueDiffuse \
    tcga/last.ckpt --local-dir inference/

3. Run inference

conda activate diff   # PyTorch 2.0.1 + CUDA 11+

python inference_heteroTissueDiffuse_camelyon.py \
    --normal_prompt  inference/promptNormal2.png \
    --tumor_prompt   inference/promptTumor2.png \
    --segmentation_root inference/masks \
    --ckpt_path      inference/epoch=000064.ckpt \
    --output_dir     outputs/inference_results

Inputs:

• --normal_prompt / --tumor_prompt: small PNG crops of representative tissue regions (provided as examples in the GitHub repo)
• --segmentation_root: folder of .npy binary masks (512×512, dtype bool, 0=normal, 1=tumor)

Outputs (per mask):

• frame_XXX.png — generated histopathology image
• prompt_frame_XXX.png — visualization of the conditioning (mask + overlaid crops)

---

Performance

Downstream Segmentation (IoU)

Training data	Camelyon16	PANDA
Real images	0.72	0.96
Synthetic (ours)	0.71	0.95
Synthetic (no conditioning)	0.51	0.82

Pathologist Assessment

A certified pathologist evaluated 120 images in a blinded study. Synthetic images conditioned with visual prompts received quality scores indistinguishable from real images:

"The generated images tended to have equal or higher quality than the real images."

---

Intended Use

• Research: generating large annotated synthetic histopathology datasets for downstream model training
• Augmentation: expanding small annotated datasets with realistic diverse tissue variations
• Privacy-preserving data sharing: synthetic data as a substitute for patient slides
• Education: illustrating tissue morphology variations

---

Training Details

Camelyon16 Checkpoint

• Dataset: Camelyon16 (lymph node whole-slide images, binary tumor/normal segmentation)
• Patch size: 256×256 pixels at 0.5 µm/px
• Training steps: 64 epochs
• Optimizer: Adam, lr=1e-6
• Hardware: A100 GPU
• Framework: PyTorch 2.0.1 + pytorch-lightning 1.4.2

Self-Supervised Extension (TCGA)

Patches from 11,765 TCGA whole-slide images were embedded using a histopathology foundation model (PathDino), then clustered into 100 tissue phenotypes via k-means. These clusters form pseudo-semantic maps for training without manual annotation.

---

Citation

@InProceedings{Alfasly2025HeteroTissueDiffuse,
    author    = {Alfasly, Saghir and Uegami, Wataru and Hoq, MD Enamul and Alabtah, Ghazal and Tizhoosh, H.R.},
    title     = {Semantic and Visual Crop-Guided Diffusion Models for Heterogeneous Tissue Synthesis in Histopathology},
    booktitle = {Neural Information Processing Systems (NeurIPS)},
    month     = {December},
    year      = {2025}
}

---

License

This model is released under the CreativeML Open RAIL-M license, inherited from CompVis/stable-diffusion. This license permits research and commercial use but prohibits use cases that cause harm (e.g., generating deceptive or malicious content). See the full license here.