Add model card, link to paper and code

#2
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +54 -0
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ pipeline_tag: image-to-image
4
+ ---
5
+
6
+ # ReChannel: Pixel-Space Dense Prediction with Text-to-Image Models
7
+
8
+ This repository contains the weights for **ReChannel** presented in [From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models](https://huggingface.co/papers/2607.06553).
9
+
10
+ ReChannel reads dense prediction targets (such as depth, surface normals, matting, and referring segmentation) out of a FLUX-Klein text-to-image DiT with lightweight per-task LoRA adapters on the (otherwise frozen) backbone.
11
+
12
+ * **Code:** [GitHub Repository](https://github.com/xmz111/ReChannel)
13
+
14
+ ## Quick Start (Inference)
15
+
16
+ To run the model locally, please clone the GitHub repository and install the dependencies:
17
+
18
+ ```bash
19
+ git clone https://github.com/xmz111/ReChannel.git
20
+ cd ReChannel
21
+ pip install -r requirements.txt
22
+ ```
23
+
24
+ You can then run dense prediction tasks on a single image:
25
+
26
+ ```bash
27
+ python infer.py --image assets/demo_input.jpg \
28
+ --tasks depth,normal,matting,refseg \
29
+ --phrase "the right couch" \
30
+ --out out.png
31
+ ```
32
+
33
+ - `--tasks`: any subset of `depth, normal, matting, refseg`.
34
+ - `--phrase`: the referring expression used by `refseg` (text-conditioned).
35
+
36
+ ## How it works
37
+
38
+ ```
39
+ RGB --VAE encoder--> latent tokens --DiT (frozen θ + task LoRA Δt, σ=0)--> token field Z_t
40
+ -- Ŷ = reshape( W_t · z_ij + b_t ) ∈ R^{p×p×K} --tile over the plane--> dense field
41
+ ```
42
+
43
+ The backbone is frozen; only a lightweight per-task LoRA adapter and the token-local linear head are trained. The head has no spatial mixing — all spatial structure comes from the adapted token field, not the head.
44
+
45
+ ## Citation
46
+
47
+ ```bibtex
48
+ @article{wang2026rechannel,
49
+ title={From RGB Generation to Dense Field Readout: Pixel-Space Dense Prediction with Text-to-Image Models},
50
+ author={Wang, Zanyi and Lin, Xin and Li, Haodong and Jiang, Dengyang and Li, Yijiang},
51
+ journal={arXiv preprint arXiv:2607.06553},
52
+ year={2026}
53
+ }
54
+ ```