FinModernBERT-large-DAPT

ModernBERT-large domain-adapted to finance with 5.65B tokens of masked-language-model pretraining. This is the Stage-1 foundation of a pipeline whose final embedding model, FinModernBERT-embed-large-v1, beats the 7B Fin-E5 on FinMTEB Summarization (+0.109) and STS (+0.010) at 1/18th the size — and this checkpoint is also the WiSE-FT interpolation anchor that made that release possible. That final model is now on the official FinMTEB leaderboard: STS #2 of 18 (top embedding model on the board) and Summarization #2, see §2.

This card documents the full project journey — goal, every stage, every measured number — with this checkpoint's role in it.

1. The project goal

Build the strongest defensible finance embedder on a single ModernBERT-large (395M), evaluated on the full English FinMTEB benchmark (35 tasks / 7 types) against the published SOTA, Fin-E5 (finance-adapted e5-mistral-7B; paper). Target: win STS and Summarization, make Retrieval competitive. "Defensible": every stage trains behind a hard decontamination gate against all FinMTEB-EN eval sets.

2. Where the pipeline ended up (context for this checkpoint)

Final model results, full FinMTEB-EN 35 tasks (see the embed card for per-task detail):

Task type	Final model (395M)	Fin-E5 (7B)	Δ
Summarization	0.588	0.480	+0.109 👑
STS	0.444	0.434	+0.010 👑
Reranking	0.961	0.990	−0.028
Clustering	0.513	0.565	−0.052
Classification	0.623	0.757	−0.133
PairClassification	0.617	0.801	−0.184
Retrieval	0.502	0.711	−0.208
Overall (7-type mean)	0.607	0.677	−0.070

Official FinMTEB leaderboard placement (EN board, 18 models, as of 2026-07-19): the final model ranks STS #2 (the top embedding model — only a lexical bag-of-words baseline scores higher; ahead of Fin-E5, voyage-3-large and NV-Embed v2), Summarization #2 (behind only voyage-3-large, with the board's single best FINDsum score, 0.745), and #11 overall. Only four models on the board hold two or more top-2 task-type finishes — voyage-3-large, text-embedding-3-large, Fin-E5, and this pipeline's 395M open-weights model.

And the journey milestone table:

Milestone	STS	Summarization	Retrieval
Stage 0: untrained ModernBERT-large	0.450	0.118	0.051
Stage 1: + 5.65B-token DAPT (this model)	0.446	0.120	0.056
+ chunk+mean-pool doc embedding (no training)	0.446	0.305	0.056
Stage 3: + 270k-pair contrastive (pure)	0.371	0.625	0.569
+ WiSE-FT α=0.65 (released embed model)	0.444	0.588	0.502
Fin-E5 7B (the bar)	0.434	0.480	0.711

3. This training run (from the actual logs)

Base	answerdotai/ModernBERT-large (395M)
Objective	Masked LM, mask rate 0.30 (ModernBERT's own rate)
Corpus	5,520,194 packed windows × 1,024 tokens ≈ 5.65B tokens finance text (SEC filings, financial news, finance web), packed from a 185 GB raw pool after cleaning + decontamination
Epochs / steps	1 epoch, 43,127 steps (128 windows ≈ 131k tokens per step)
LR schedule	peak 5e-5, ~2.3k-step warmup, cosine decay to ~0
Precision / memory	bf16, gradient checkpointing, full finetune (no LoRA)
Hardware	2× RTX 3090 (24 GB), DDP
Loss trajectory	1.134 (init) → 1.042 @1k → 0.951 @3k → 0.907 @7k → 0.888 final (1k-step window means); grad-norm stable ≈0.5–0.9

Decontamination (before any training)

Every FinMTEB-English eval text was indexed with a word-shingle overlap index; corpus records overlapping any eval set, eval-source datasets, and test/dev splits were dropped; a hard assert_clean gate refuses to emit a corpus otherwise. Known-contaminated lineages (FiQA/ConvFinQA etc.) excluded outright. The corpus is not released (source licensing does not permit redistribution).

4. What DAPT does — and doesn't do (measured honestly)

Scored on FinMTEB-EN (15-task subset: 2 STS + 3 Summ + 10 Retrieval), mean pooling:

task type	stock ModernBERT-large	this model	Δ	Fin-Retriever-base	Fin-E5
STS	0.450	0.446	−0.005	0.294	0.434
Summarization	0.118	0.120	+0.002	0.281	0.480
Retrieval	0.051	0.056	+0.005	0.402	0.711

DAPT alone barely moves zero-shot embedding scores — an MLM objective doesn't reshape sentence geometry. Its measured value in this project came from two places:

1. Foundation for contrastive training. Stage 3 (270,238 synthetic + open pairs, positive-aware hard negatives mined with e5-large-v2, CachedMNRL scale 20 + CoSENT, 2 epochs on 2×3090) was trained from this checkpoint and reached Retrieval 0.569 / Summarization 0.625 (pure).
2. WiSE-FT interpolation anchor. Pure contrastive training collapsed STS 0.450 → 0.371, failing the pre-registered release gate (≥0.434 = Fin-E5 parity). The fix: w = α·contrastive + (1−α)·this_checkpoint:

α (contrastive share)	STS	Summarization	Retrieval	gate ≥0.434
1.00 (pure contrastive)	0.371	0.625	0.569	✗
0.80	0.402	—	—	✗
0.65 (released)	0.444	0.588	0.502	✓
0.50	0.465	0.457	0.327	✓ (retrieval collapses)
0.30	0.446	—	—	✓
0.00 (this model)	0.446	0.305*	0.056	✓

* with chunk+mean-pool doc-side embedding. Note α=0.5 beats both endpoints on STS — the classic WiSE-FT effect. A finance-DAPT anchor made the interpolation safe: general knowledge returns without leaving the finance domain.

5. The rest of the pipeline (what was built on top)

• Stage 2 — synthetic pairs: an open-weights 35B MoE teacher (AWQ/vLLM) over SEC chunks → 115,238 filtered finance pairs incl. 16,696 doc↔summary positives + 16,700 mismatched score-0 summary negatives (the lever Fin-E5 never trained — its own adaptation regressed Summarization 0.528→0.480) + 1,452 term→definitions; blended with open general data to 270,238 pairs (59% finance).
• Long-doc discovery: chunk+mean-pool doc embedding = Summarization 0.120→0.305 with zero training (FNS docs average ~70k+ tokens; head-truncation was destroying signal).
• Stage 3 — contrastive: 2 epochs / 1,976 steps, effective 256 anchors/step, 255 in-batch negatives/device, lr 2e-5 cosine, bf16, ~8h on the same 2×3090 pair.

6. Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

tok = AutoTokenizer.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")
model = AutoModelForMaskedLM.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")

text = "The company's [MASK] margin expanded 150 basis points year-over-year."
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
mask_idx = (inputs.input_ids == tok.mask_token_id).nonzero()[0, 1]
print(tok.decode(logits[0, mask_idx].topk(5).indices))

As a fine-tuning base (recommended use):

from transformers import AutoModel
encoder = AutoModel.from_pretrained("BatuhanECB/FinModernBERT-large-DAPT")  # drops MLM head

Native context 8,192 tokens (ModernBERT); DAPT windows were 1,024 tokens. English, finance-domain, Apache-2.0.

7. Limitations

• Not an embedding model out of the box — zero-shot sentence geometry ≈ the untrained base (see §4). Use the embed model for embeddings, or fine-tune this.
• English-only, finance-domain MLM; general-domain MLM quality may have drifted.
• Corpus not released (licensing); decontamination applies to FinMTEB-EN specifically.

Related

• FinModernBERT-embed-large-v1 — the final contrastive embedding model built from this checkpoint.
• FinModernBERT-pairs-sec-synthetic-v1 — the released 115k SEC-synthetic contrastive pair set used in Stage 3.
• ModernBERT-large — base model.
• FinMTEB — benchmark (paper).