SelfCite-8B-CC-SFT

Paper: SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models
Authors: Yung-Sung Chuang†, Benjamin Cohen-Wang†, Shannon Zejiang Shen†, Zhaofeng Wu†, Hu Xu‡, Xi Victoria Lin‡, James Glass†, Shang-Wen Li‡, Wen-tau Yih‡
† Massachusetts Institute of Technology, ‡ Meta AI

This is the intermediate ContextCite supervised fine-tuning (CC-SFT) checkpoint used in the fully self-supervised setting of the SelfCite paper. It is initialized from Llama-3.1-8B-Instruct and trained on SFT data generated with ContextCite using 256 ContextCite calls.

This checkpoint is the model before SimPO preference optimization. For the resulting SelfCite model after SimPO, use voidism/SelfCite-8B-from-CC.

Please refer to the SelfCite repository for training and usage details.

Training details

• Base model: meta-llama/Llama-3.1-8B-Instruct
• Training examples: 11,185
• Learning rate: 7e-6
• Epochs: 1
• Per-device batch size: 1
• Number of devices: 8
• Effective batch size: 8
• Scheduler: cosine with a 0.1 warmup ratio
• Precision: bfloat16
• Transformers: 4.44.2
• PyTorch: 2.4.1+cu121

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "voidism/SelfCite-8B-CC-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Citation

Please cite SelfCite and LongCite if they are helpful to your work.

@inproceedings{chuang2025selfcite,
  title={SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models},
  author={Yung-Sung Chuang and Benjamin Cohen-Wang and Shannon Zejiang Shen and Zhaofeng Wu and Hu Xu and Xi Victoria Lin and James Glass and Shang-Wen Li and Wen-tau Yih},
  journal={arXiv preprint arXiv:2502.09604},
  year={2025}
}

@article{zhang2024longcite,
  title={LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA},
  author={Jiajie Zhang and Yushi Bai and Xin Lv and Wanjun Gu and Danqing Liu and Minhao Zou and Shulin Cao and Lei Hou and Yuxiao Dong and Ling Feng and Juanzi Li},
  journal={arXiv preprint arXiv:2409.02897},
  year={2024}
}