manifesta/adaption_verified_math_code_instruct

Show Your Work: Verified Math and Code (Gemma 4 31B LoRA)

A PEFT LoRA adapter trained on 17,586 math and code problems where the answers were independently checked: 9,104 math rows carry a gold answer computed separately, 4,156 code rows were executed against unit tests, and 3,740 general rows were deliberately left unverified as an anti-forgetting slice.

It lost. The head-to-head result is 46 wins for the adapted model against 54 for the base. That is a regression, not a tie and not a win. Training loss fell from 3.48 to 1.24 and held-out eval loss fell from 2.37 to 1.22, and the model still came out behind.

This card gives the diagnosis from the weights and the training state rather than dressing the number up. There are two causes and they compound: the base was already strong on this domain, and the run was numerically unstable, with 29 of 59 optimizer steps hitting the gradient clip.

Contents

• TL;DR
• Quickstart
• What is actually in these weights
• Evaluation
• Why it lost
• Numerical stability
• The cross-run finding
• Training
• Intended use and limits
• Dataset
• Live interface
• Related models
• License
• Citation

TL;DR

• What it is: a rank-8 LoRA adapter on google/gemma-4-31B-it, 10,956,800 trainable parameters, 43.9 MB.
• What it was for: step by step derivations for math and code, with the working shown rather than just the answer.
• The number: 46 for the adapted model, 54 for the base. A regression of 8 points.
• The reasons: the base scored 8.0 out of 10 on the source corpus before any adaptation, the highest starting point of the four datasets in this project, so there was almost nothing to add. On top of that, 29 of 59 steps exceeded the clip threshold of 2, peaking at 677.
• Good news on loading: unlike the sibling adapters in this project, the declared base model string is correct and resolves. google/gemma-4-31B-it is ungated and Apache-2.0.
• Built by: MANIFESTA (Aivaras Navardauskas) for the Adaption AutoScientist Challenge, Math and Code.

Quickstart

The base string in adapter_config.json is google/gemma-4-31B-it, which resolves, is ungated, and is Apache-2.0 licensed. No workaround needed here. (The two chart adapters in this project both shipped broken base strings, so this is worth stating explicitly.)

Gemma 4 is a conditional-generation checkpoint with a vision tower, so use the image-text-to-text auto class even when you only want text.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

BASE = "google/gemma-4-31B-it"
ADAPTER = "manifesta/adaption_verified_math_code_instruct"

processor = AutoProcessor.from_pretrained(BASE)
base = AutoModelForImageTextToText.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

messages = [{
    "role": "user",
    "content": [{"type": "text", "text":
        "A factory makes 1,440 pencils in 8 hours. Two machines are down, cutting output "
        "by 40%. How many pencils are made in a 6 hour shift? Show your work."}],
}]

inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Because the base string is correct, the one-line loader works too:

from peft import AutoPeftModelForCausalLM
model = AutoPeftModelForCausalLM.from_pretrained("manifesta/adaption_verified_math_code_instruct")

Optional merge:

model = model.merge_and_unload()

Given the evaluation result, the most useful thing to do with this snippet is run the same prompt through the base with and without the adapter attached and compare. That is what the judge did, and the base won.

What is actually in these weights

Read straight out of adapter_model.safetensors and adapter_config.json.

Property	Value
PEFT type	LoRA, task_type: CAUSAL_LM
Rank r	8
lora_alpha	8 (scaling factor alpha/r is exactly 1.0)
lora_dropout	0.0
bias	none
use_rslora / use_dora	false / false
Target modules	q_proj, v_proj
exclude_modules	empty
Tensors in the file	220
Decoder layers touched	60 of 60 for q_proj, 50 of 60 for v_proj
Vision tensors	0
Trainable parameters	10,956,800 (float32, 43.9 MB)
Share of the 31B base	roughly 0.035%

The v_proj gap is architecture, not a bug

v_proj is missing on exactly ten layers: 5, 11, 17, 23, 29, 35, 41, 47, 53 and 59. Every sixth layer.

Those are precisely the layers listed as full_attention in the base model's layer_types. The other fifty are sliding_attention. Checking the base model's own weight map confirms it: model.layers.5.self_attn contains q_proj, k_proj, o_proj, q_norm and k_norm, and no v_proj.weight at all, while model.layers.4.self_attn does have one.

The reason is in the base config: attention_k_eq_v: true, with num_global_key_value_heads: 4 and global_head_dim: 512 on the global layers. On full-attention layers the key and value projections are shared, so there is no separate v_proj module for PEFT to wrap. It asked for v_proj on all 60 layers and silently got it on 50.

Nothing failed. But it does mean the adapter covers less of the attention stack than target_modules: [q_proj, v_proj] suggests, and the layers it covers least are the global-attention layers, which are the ones carrying long-range context. For multi-step derivations, long-range context is the part that matters most.

No vision tensors

Gemma 4 31B is multimodal, with a 27-layer vision tower. This adapter touched none of it. For a text-only math and code task that is the right call, so it is recorded here for completeness rather than as a criticism. It is a different story on the chart QA adapter, where the frozen vision tower is the whole explanation for that run's null result.

Evaluation

Judged by Adaption's AutoScientist evaluator on a held-out split, against the same base model it was trained from. Head-to-head, blind.

Metric	Base	Adapted	Change
Win rate	54	46	-8 points
Held-out eval loss (first eval)	2.3657
Held-out eval loss (final)		1.2200	-1.146
Training loss	3.482 (step 1)	1.238 (step 59)	-2.244

Eval loss across the five checkpoints: 2.3657, 1.5618, 1.3206, 1.2417, 1.2200.

Read those two rows together, because they disagree. Loss dropped by 64% and the model still lost the head-to-head by 8 points. A falling loss curve says the adapter learned to reproduce the training distribution. It says nothing about whether reproducing that distribution is an improvement over what the base already did. Here it was not.

This is the single most useful thing in this repo: eval loss is not the headline metric, and if you optimise against it you can ship a regression with a beautiful chart attached.

Why it lost

Two causes, and they compound.

1. The base was already good, so there was no room. Adaptive Data scored the source corpus at 8.0 out of 10 before adaptation, grade B, the highest starting quality of the four datasets built in this project. Adaptation moved it to 8.2, a +2.5% lift, and the grade did not change. Compare the agronomy corpus at 5.0 to 8.4 (+68.0%) and the chart corpus at 6.0 to 7.1 (+18.3%).

Gemma 4 31B is instruction-tuned and already writes competent step by step derivations for exactly this kind of problem. A rank-8 adapter on two projections, roughly 0.035% of parameters, cannot add mathematical capability the base lacks. What it can do is change style and formatting. When the base's existing style is already good, changing it is as likely to hurt as help, and a blind judge comparing two competent derivations will punish any drift toward the training corpus's quirks.

2. Two thirds of the updates were clipped. See below.

3. The global-attention layers got the least adaptation. As covered above, the ten full_attention layers received q_proj only. Those are the layers handling long-range dependency, which is what a multi-step derivation depends on to keep an intermediate result consistent from line four to line twelve.

What would need to change: a lower learning rate or a longer warmup to stop the gradient spikes, a higher rank if the goal is genuinely new capability rather than style transfer, and a harder corpus. Problems the base already solves at 8.0 out of 10 are the wrong training signal. The value should sit in problems it gets wrong.

Numerical stability

This run was not stable.

Gradient norm statistic	Value
Clipping threshold (max_grad_norm)	2
Peak gradient norm	677.21 (step 38)
Median gradient norm	1.97
Minimum	0.389
Steps above 1	39 of 59 (66%)
Steps above the clip threshold of 2	29 of 59 (49%)
Steps above 10	9 of 59 (15%)
Steps above 100	5 of 59 (8%)

Largest spikes: step 38 at 677.2, step 15 at 645.0, step 40 at 389.4, step 14 at 172.5, step 26 at 138.6. The final step of the run, step 59, was still at 40.1.

What clipping at 2 does when the true norm is 677 is keep the direction and throw away the magnitude. The update becomes a fixed-size step along whatever direction that batch happened to point in, and the relative weighting between batches is gone. When that happens on half the steps of a 59-step run, the optimizer is not really following the loss surface any more, it is taking a sequence of equal-length steps in noisy directions.

The loss still came down, which is worth sitting with. A falling loss curve does not certify a healthy run.

For contrast, the chart QA adapter trained on the same platform with the same clip threshold peaked at 0.72 and clipped on zero of its 34 steps. Same settings, completely different numerical behaviour, which points at the data and the base rather than the configuration.

(The Adaption run dashboard reports the peak as 744.9. The 677.21 above is the maximum in trainer_state.json in this repo, which is the number you can verify yourself. Both are two to three orders of magnitude over the threshold and the conclusion is the same.)

The cross-run finding

Three completed AutoScientist runs, three different base models. Win rate tracked how weak the base already was on the domain, and did not track dataset size.

Run	Base	Params	Rows ingested	Steps	Rows per step	Win rate (adapted vs base)
Chart QA, first build	Qwen3.5 9B	9B	6,976	21	332	51 vs 49
Chart QA 17k	Gemma 3 27B	27B	17,070	34	502	50 vs 50
Verified math and code (this model)	Gemma 4 31B	31B	17,586	59	298	46 vs 54

The bigger the base, the smaller the win. 9B scored 51, 27B scored 50, 31B scored 46. This model sits at the bottom of that trend and it is the only one that went backwards.

Scaling the corpus did nothing. The chart corpus was rerun at 2.4x the rows, 6,976 to 17,070, and the win rate went 51 to 50. Because batch_size is set to max, extra rows widen the batch rather than adding updates: step count went 21 to 34 while rows per step went 332 to 502. More rows per step is a smoother gradient estimate for the same small number of updates, not more learning.

Caveat, stated plainly: three data points, three different base models, three different corpora, three different domains. Base size, base family and task difficulty all move together, so nothing here isolates a cause. It is not a controlled experiment and should not be read as one. Testing it properly means holding the corpus fixed and varying only the base.

Training

Trained with Adaption AutoScientist, SFT with LoRA, on Adaptive Data output.

Hyperparameter	Value
Base model	google/gemma-4-31B-it (resolves correctly)
Method	SFT, LoRA (PEFT), chat format
LoRA rank / alpha / dropout	8 / 8 / 0.0
Trainable modules	q_proj, v_proj (requested on 60 layers, attached on 60 and 50)
Learning rate	1e-4
Scheduler	cosine, 0.5 cycles, warmup ratio 0.1, min LR ratio 0.1
Weight decay	0
Max gradient norm	2
Epochs	1
Batch size	max (per-device train batch size 1 with accumulation)
Optimizer steps	59
Evaluations	5, every 11 steps
Train on inputs	false
Precision	bfloat16 base, float32 adapter weights
Total FLOPs	4.073e18
Adaption job ID	9f46e2b9-ac73-4b97-ad13-cc6ccd1501cf
Training experiment ID	3ce14a29-6d5d-48ea-849f-372a6102c360
Adaption dataset ID	6ad1a83f-b806-4a20-ba6d-239029d30711

Intended use and limits

Use it for

• Reproducing and inspecting this regression. That is the honest primary use.
• Studying the gap between a falling loss curve and a falling win rate, which this run demonstrates about as cleanly as it can be demonstrated.
• Studying gradient instability in short LoRA runs, using the 59 logged steps in trainer_state.json.
• A baseline for a rerun with a lower learning rate, a longer warmup, or a harder corpus.

Do not use it for

• Production math or code assistance. Stock google/gemma-4-31B-it beat it 54 to 46 on the held-out split. Use the base.
• Anything where a wrong derivation carries cost: engineering, finance, dosing, safety calculations. Verified training data does not make outputs verified.
• Executing generated code without a sandbox. The training data was executed against unit tests; the model's output at inference time was not.

Technical limits

• The adapter is behind its own base model on the only head-to-head evaluation that was run. Treat the weights as a research artefact.
• Rank 8 on two projections, roughly 0.035% of base parameters, one epoch, 59 optimizer steps.
• v_proj is absent on the ten global-attention layers. Not a defect, but the coverage is thinner than the config implies.
• Half the optimizer steps were clipped. The weights reflect a run that did not converge cleanly.
• English only.
• Adapter weights are float32 while the base is bfloat16. PEFT casts on load, but the adapter file is double the size a bf16 export would be.
• One judge, one held-out split. Reported as-is, no reruns, no best-of selection.

Dataset

Trained on Adaption dataset 6ad1a83f-b806-4a20-ba6d-239029d30711, built from manifesta/verified-math-code-17k (also on Kaggle).

• 17,586 rows ingested, expanded to 21,063 rows in the training split after adaptation
• 9,104 math rows with an independently computed gold answer
• 4,156 code rows executed against unit tests, 100% of them run
• 3,740 general instruction rows deliberately left unverified, as an anti-forgetting slice
• Domain mix after adaptation: math 56%, code 25%, science 3%, with a long tail across roughly 40 further domains at 1% or below

Adaptive Data lifted the corpus quality before training:

Adaptive Data metric	Before	After
Quality score	8.0	8.2 (+2.5%)
Grade	B	B
Percentile	15.3	17.8

That +2.5% is the smallest lift of the four datasets built in this project, and it is the clue. The corpus started at 8.0, so there was very little for adaptation to fix, and by extension very little for the model to learn that it did not already know. +2.5% is an improvement in the data, not in the model. The model's own number went the other way.

Live interface

The full written case for this entry, every claim paired with the command that checks it, is at The Verdict.

Show Your Work runs at manifesta.adaptionlabs.app. Give it a problem, get a step by step derivation with a verification panel, and use the "Change a number" button to check that the derivation actually recomputes rather than pattern matching to a memorised answer.

Related models

• manifesta/adaption_scientific_chart_qa_17k, scientific chart QA on Gemma 3 27B. 50 vs 50, a null result, because the LoRA covered zero vision tensors while the task is entirely visual. Its declared base string 404s.
• manifesta/scientific-chart-qa-lora-qwen3.5-9b, the first chart build on a 9B base. 51 vs 49, 21 optimizer steps, LoRA on only 8 of 32 decoder layers and again zero vision tensors. Its declared base togethercomputer/Qwen3.5-9B also 404s.
• manifesta/brandvoice-marketing-model, the Part 1 marketing adapter that did work, 56% win rate on Llama 3.3 70B. Style transfer on a base with a weak default style, which is the case where a small adapter has room to move.

Everything behind these weights is public

The dataset this adapter was trained on, the scripts that built it, and a verifier that rechecks every number claimed here against the live artifacts:

https://github.com/A1VARA5/verified-math-code-17k

git clone https://github.com/A1VARA5/verified-math-code-17k
cd verified-math-code-17k
python verify.py

Standard library only, no install step and no account. 23 checks, and the same 23 run on a daily schedule in GitHub Actions, so the badge above goes red if any claim on this card stops being true. The tensor facts in this card are among the checks: layer coverage is re-derived from the published adapter_model.safetensors by name, not copied from the config.

Dataset: https://huggingface.co/datasets/manifesta/verified-math-code-17k

License

• Adapter weights: Apache-2.0.
• Base model: google/gemma-4-31B-it, Apache-2.0 and ungated. No access request needed.
• Training dataset: manifesta/verified-math-code-17k, CC0-1.0.

Citation

@misc{showyourwork_mathcode_2026,
  title  = {Show Your Work: a verified math and code adapter for Gemma 4 31B, and why it regressed},
  author = {Navardauskas, Aivaras},
  year   = {2026},
  note   = {MANIFESTA. Adapted with Adaptive Data by Adaption, AutoScientist Challenge, Math and Code. Win rate 46 vs 54. Half the optimizer steps clipped.},
  howpublished = {\url{https://huggingface.co/manifesta/adaption_verified_math_code_instruct}}
}

Built with Adaptive Data by Adaption. Platform documentation at docs.adaptionlabs.ai. Dataset, adapter and analysis by MANIFESTA (Aivaras Navardauskas) for the AutoScientist Challenge.