BAM-B0

BAM-B0 (BabyLM Aligned MNTP) is a 96.3M-parameter GPT-BERT/MNTP masked language model trained for the BabyLM 2026 Strict track.

Model

• Architecture: custom GPT-BERT masked language model
• Parameters: 96,333,280
• Tokenizer: 16k BabyLM Strict tokenizer
• Objective: 15 masked-next-token-prediction steps to 1 causal language modeling step
• Initialization: random initialization

Evaluation

The B0-only Strict submission artifact contains final predictions, B0's own 28-checkpoint fast-evaluation trajectory, GlobalPIQA, filtered Entity Tracking predictions, Reading predictions, and 28-stage AoA surprisals. It was checked with the 2026 leaderboard validation code. The JSON is retained separately for the official BabyLM submission and is intentionally not included in this model repository.

The model uses custom configuration and modeling files included in this repository; load it with trust_remote_code=True.