ctogaurav/GLM_OCR

GLM-OCR: Fine-tuned for Handwritten Math OCR → LaTeX

LoRA fine-tunes of zai-org/GLM-OCR (0.9B) for transcribing handwritten university-level math answer sheets directly into compilable LaTeX.

Code: github.com/realgauravvyas/ocr2tex — full data pipeline, GLM-OCR + Baidu OCR training scripts, benchmark harness, and the live dashboard.

Two adapter versions are included in this repo:

Version	Path	Notes
v3.1	v3.1/	Trained on 4,672 pages
v4.1	v4.1/	Trained on 12,575 pages; warm-started from the v4 adapter

Results (held out, 700 pages)

CER is measured against silver labels from a teacher VLM, not human-verified ground truth — treat absolute CER as teacher-agreement, not true transcription accuracy. Relative comparisons across rows share the same reference and remain sound.

Results are grouped by comparison type, since mixing them in one flat table obscures which rows are actually comparable:

Out-of-the-box baselines (neither system fine-tuned on this project's data)

System	Mean CER	Med CER	Compile %	Struct %	BLEU-4	chrF	MathF1	Latency (s)
Base GLM-OCR (frozen)	0.5151	0.4609	0.0	0.0	0.4583	0.5370	0.7031	8.38
Baidu OCR (stock)	0.7176	0.5768	42.7	77.6	0.3113	0.4792	0.6264	40.79

Fine-tuned on the same 12,575-page training corpus (workspace/9_split) — controlled comparison

System	Mean CER	Med CER	Compile %	Struct %	BLEU-4	chrF	MathF1	Latency (s)
Baidu OCR (fine-tuned)	0.4071	0.3523	62.7	71.7	0.5872	0.7070	0.7959	24.93
Baidu OCR (fine-tuned v2)	0.4258	0.3501	64.9	71.9	0.5958	0.7176	0.7983	29.00
GLM-OCR v4 (ours, intermediate)	0.3837	0.2834	81.3	90.4	0.6489	0.7504	0.8277	15.62
GLM-OCR v3.1 (ours)	0.3971	0.2957	88.9	91.4	0.6180	0.7093	0.8171	14.12
GLM-OCR v4.1 (ours)	0.3816	0.2783	82.4	90.9	0.6513	0.7531	0.8272	13.44

Note: v3.1 was trained on a 4,672-page subset of this corpus, not the full 12,575 (see Training configuration below) — it's grouped here because it's still a same-source fine-tune comparison, not an out-of-the-box baseline.

• Base compile rate is exactly 0.0% — 0 of 700 pages produced a compiling PDF without fine-tuning.
• v4.1 has the lowest CER and best BLEU/chrF/MathF1, but v3.1 compiles more often (88.9% vs 82.4%). Scaling the training set 4,672 → 12,575 pages bought only ~0.016 CER — evidence the task is annotation-bound, not capacity-bound. The compile-rate regression tracks a shift in average output length (0.98x → 1.07x of reference length in v4.1).
• Both Baidu fine-tunes were trained on the identical page corpus used for GLM-OCR v4/v4.1, so the gap between them and v3.1/v4/v4.1 reflects the base models and training recipes, not a data advantage on either side.
• Headline: mean CER 0.515 → 0.382, a 25.9% relative reduction over the frozen base model.

Training configuration

This repo ships the two versions selected from a broader set of experiments (v3, v3.1, v4, v4.1) run over the course of the project. Full training configs are documented below for v3.1 and v4.1 only — the two adapters actually released here. v4 appears in the results table above as an intermediate checkpoint for context, but its run wasn't logged with the same rigor as the two released versions and isn't included as a downloadable adapter in this repo.

Both released versions share the same LoRA setup — r=32, alpha=64, dropout=0.05, targets q_proj/k_proj/v_proj/o_proj/gate_proj/up_proj/down_proj — and were trained on a single RTX 3060 12GB (no cloud compute) with Python 3.11.9, PyTorch 2.10.0+cu130, Transformers 5.9.0, PEFT 0.18.1, epochs=2, batch=1, grad_accum=8 (effective batch 8), bf16, seed=42. They differ as follows:

	v3.1	v4.1
Training pages	4,672	12,575
Full validation partition	584	698
Periodic in-training eval subset	200 (capped, for speed)	200 (capped, for speed)
Learning rate	2e-5	1e-5
Warm start	from the v3 adapter	from the v4 adapter
Steps	1,168	3,144
Best val loss	0.108	0.164

• max_length=3584, max_image_tokens=1536
• Decoding (benchmark): greedy (do_sample=False), max_new_tokens=2048, repetition_penalty=1.0
• Training time: v3.1 ~5 hours (1,168 steps) · v4.1 ~12.7 hours (3,144 steps, run resumed across multiple sessions on shared personal hardware)

Data

Raw scans → blank/printed filter → deskew/prepare → teacher-VLM annotation → automated quality review → pdflatex-validity filter → train/val/test split (12,575 / 698 / 700).

Usage

Load either version by pointing subfolder at v3.1 or v4.1:

from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

base = "zai-org/GLM-OCR"
processor = AutoProcessor.from_pretrained(base, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(base, trust_remote_code=True)

# v4.1 — lower CER, best BLEU/chrF/MathF1
model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v4.1")

# v3.1 — higher compile rate (88.9% vs 82.4%)
# model = PeftModel.from_pretrained(model, "ctogaurav/GLM_OCR", subfolder="v3.1")

Running locally (LM Studio / Ollama / llama.cpp)

v4.1 is available as a ready-to-download GGUF, verified working end-to-end (not just "conversion succeeded" — output checked against this project's own validated ground truth): ctogaurav/GLM_OCR-GGUF. Download both GLM-OCR-v4.1-Q8_0.gguf and mmproj-GLM-OCR-v4.1-Q8_0.gguf (~1.2GB total) into the same folder in LM Studio, or run directly with llama.cpp's llama-mtmd-cli.

Producing this required a small local patch to llama.cpp's own conversion code — the public zai-org/GLM-OCR checkpoint doesn't ship the weights its conversion path expects for an unused auxiliary layer, causing a hard load failure otherwise. Full details, including the fix, are in the GGUF repo's README.

v3.1 isn't converted yet — the same process applies (merge with PeftModel.merge_and_unload(), convert with a patched llama.cpp), just not done for this version yet.

License

Both adapters are released under MIT, matching the base model's license. The base zai-org/GLM-OCR model card notes that Apache-2.0-licensed components (PP-DocLayoutV3) are used in some parts of its broader published pipeline — this project doesn't use PP-DocLayoutV3 anywhere, so only MIT applies here. You're free to merge, quantize, redistribute, and use these adapters commercially, including as GGUF conversions.

Limitations

• Evaluation CER is against teacher-VLM silver labels, not human-verified transcriptions.
• Trained on university-level handwritten math; performance on other domains (e.g. chemistry notation, diagrams, non-English handwriting) is untested.