stepfun-ai

/

Step3-VL-10B

like 410

Follow

StepFun 2.95k

Summary

Fix two Transformers 5 compatibility issues in Step3-VL:

• Rebuild EncoderRope2D.freqs_cache when it was created on the meta device. The cache is non-persistent and is therefore not restored from the checkpoint.
• Use is_first_iteration for Transformers 5 generation while retaining cache_position forwarding for Transformers 4.x.

Regular Transformers 4.x model construction continues to use the original eager RoPE cache path.

Tests

Tested locally with PyTorch 2.12.1:

• Transformers 4.57.0: 9 tests passed
• Transformers 5.13.1: 9 tests passed

The tests cover regular eager cache construction, meta-device cache rebuilding, all 47 RoPE caches in the published vision configuration, non-persistent state-dict behavior, .half() dtype conversion, first and cached generation iterations, Transformers 4.x cache_position forwarding, and Transformers 5.x is_first_iteration behavior.