tencent/Rosetta-inference

🪨 Rosetta: Composable Native Multimodal Pretraining

Escaping the Forgetting-Synergy Dilemma in Native Multimodal Pretraining

Xiangyue Liu¹, Zijian Zhang², Miles Yang², Zhao Zhong², Liefeng Bo², Ping Tan^1*

¹HKUST    ²Tencent Hunyuan

Figure 1. (Left) Performance on MMLU (language ability) across composable pretraining stages (LM → +MMU → +T2I). Standard MoE and structurally-isolated MoT suffer catastrophic routing collapse upon integrating continuous generative objectives. Rosetta maintains a stable semantic anchor throughout all stages. (Right) Qualitative image generation results from the Rosetta model.

🏗️ Architecture

Figure 2. Rosetta FFN. Three mechanisms enable non-destructive modality expansion: (1) Unified Attention — globally shared QKV projections preserve dense cross-modal interactions. (2) Composable FFN — modality-specific plug-and-play experts (Text / ViT / VAE) are bridged by a single Global Shared Expert that anchors foundational knowledge. (3) Conflict-Free Optimization (MAOP) — surgically neutralizes destructive gradients with zero memory overhead.

📊 Benchmarks

✍️ Citation

@misc{liu2026rosettacomposablenativemultimodal,
      title={Rosetta: Composable Native Multimodal Pretraining}, 
      author={Xiangyue Liu and Zijian Zhang and Miles Yang and Zhao Zhong and Liefeng Bo and Ping Tan},
      year={2026},
      eprint={2607.00293},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.00293}, 
}