stride-math20-a2-step400

STRIDE fine-tune of Qwen/Qwen2.5-Math-1.5B on MATH (lighteval), trained with verl. Selected as the best-validation checkpoint for this arm by measured pass@1.

IMPORTANT: do not apply a chat template

This model was trained on raw prompt text. verl's RLHFDataset has apply_chat_template=False by default and it was never enabled for these runs, so every training rollout saw the bare problem string with no system message and no <|im_start|> markers.

Qwen2.5-Math ships a chat template that injects a system prompt. Applying it at inference creates a train/eval mismatch that we measured directly: responses grow longer, truncation at 1024 tokens rises from 4.3% to 21.8%, and pass@1 drops by roughly 19 points.

from vllm import LLM, SamplingParams

llm = LLM(model="sandeep123/stride-math20-a2-step400", dtype="bfloat16", max_model_len=2048)
params = SamplingParams(n=10, temperature=1.0, top_p=1.0, top_k=-1,
                        max_tokens=1024, stop_token_ids=[151643, 151645])

# CORRECT -- raw prompt string
outputs = llm.generate([problem_text], sampling_params=params)

# WRONG -- llm.chat() applies the chat template and costs ~19 points of pass@1
# outputs = llm.chat([[{"role": "user", "content": problem_text}]], params)

The training prompt appends these instructions to each problem:

Let's think step by step, break your reasoning into numbered steps.

IMPORTANT rules:
1. You MUST produce at least 2 reasoning steps.
2. The final \boxed{X} must be on its OWN LINE, NOT part of any Step.
3. Do NOT write anything before Step 1 or after \boxed{}.

Training

setting	value
method	STRIDE
STRIDE alpha	2.0
checkpoint step	400
epochs	20 (~1160 steps)
rollouts K	6
train batch	128
max response length	1024
KL (in-reward)	0.01
sampling temperature	1.0
validation freq	every 400 steps

Validation metrics at this checkpoint

metric	value
pass@1	0.7253
pass@k (k=6)	0.9375
duplicate-opening rate	0.031
non-ASCII fraction	0.01%

Read these numbers with care. They come from the training-time validation set, which is the first 128 rows of the MATH test parquet (768 responses). Two caveats we measured:

• Small: standard error is roughly +/-4 points, so differences of a few points between arms are not meaningful on their own.
• Optimistically biased: under identical generation, that 128-prompt subset scores 0.5429 while the full 5000-prompt test set scores 0.4154 -- about 13 points easier. These validation figures are not full-test-set numbers.

Context

Part of a STRIDE (step-level diversity in RL exploration) alpha sweep on MATH. The untrained base model scores pass@1 0.4805 / pass@k 0.8672 on the same validation subset.

On MATH the step-level diversity metrics have very little headroom: the base model's duplicate-opening rate is already 0.0% (versus ~58% on ScienceQA), and an LLM judge counting distinct solution methods returns exactly 1 for ~85% of problems regardless of arm. Accuracy differences between the STRIDE arms and the GRPO baseline on this validation set are within noise.