medarc/spectra-openmidnight-merged

SPECTRA (merged) - OpenMidnight

Full-weight release of SophontAI/OpenMidnight made robust to changes in slide acquisition -- scanner, stain, centre. The SPECTRA rank-32 LoRA delta has already been folded into the base weights, so this repository is a drop-in replacement for the base model: you build the architecture and load_state_dict, and you do not need peft installed.

Training was contrastive on registered PLISM tiles -- the same physical tissue location imaged under many scanner/stain conditions -- with an objective that pulls matched conditions of one tile together while pushing different tiles apart.

If you would rather keep the base weights untouched and apply a small delta at load time, use the adapter-only release instead.

Base model. These are SophontAI/OpenMidnight weights (Apache-2.0) with a SPECTRA LoRA adapter merged in, released here by the same organisation. Apache-2.0 terms and the attribution in NOTICE apply.

Base model

Base repository	SophontAI/OpenMidnight
Pinned revision	87189e66
Loader	timm
Merged modules	attn.qkv, attn.proj, mlp.fc1, mlp.fc2 (4 per block x 40 blocks = 160 modules)
LoRA rank / alpha / scaling	32 / 64 / 2.0
Weight dtype	fp32
Embedding dimension (this readout)	1536

The delta was trained and evaluated against that exact base revision. The merge is W <- W + (alpha/r) * (B @ A), computed in fp32 before any device move or dtype cast.

Seeds

Three independent training seeds are shipped as subfolders. They are not an ensemble -- pick one, and report spread across all three.

Folder	Original training seed	Selected step	Training run
seed0/	s0	150	genMASK-c50-ms500-openmidnight-s0-t900-438670
seed1/	s1	150	genMASK-c50-ms500-openmidnight-s1-t900-438671
seed2/	s2	150	genMASK-c50-ms500-openmidnight-s2-t900-438672

Folder names are positional (seed0/1/2); the "original training seed" column records the seed label the run actually used, so provenance is not lost.

Usage

import json, timm, torch
import safetensors.torch as st
from huggingface_hub import snapshot_download

d = snapshot_download("medarc/spectra-openmidnight-merged", allow_patterns=["seed0/*"])
cfg = json.load(open(f"{d}/seed0/config.json"))

model = timm.create_model(
    cfg["architecture"],            # "vit_giant_patch14_reg4_dinov2"
    pretrained=False,
    num_classes=0, global_pool="token",
    img_size=224, init_values=1e-5, dynamic_img_size=False,
)
model.load_state_dict(st.load_file(f"{d}/seed0/model.safetensors"), strict=True)
model = model.float().eval().cuda()

Preprocessing -- this must match exactly:

import torchvision.transforms as T

# DEFAULT transform -- used for HEST, PathoROB and CPTAC in the paper.
tf = T.Compose([
    T.Resize(256, interpolation=T.InterpolationMode.BICUBIC),   # crop_pct 0.875
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

# THUNDER-ONLY transform -- squash resize, no crop. See warning 1.
tf_thunder = T.Compose([
    T.Resize((224, 224)),
    T.ToTensor(),
    T.Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225)),
])

Forward pass and readout:

from PIL import Image

img = Image.open("tile.png").convert("RGB")
x = tf(img).unsqueeze(0).cuda()
with torch.inference_mode():
    h = model.forward_features(x)     # (B, 5 + 256, 1536)
feat = h[:, 0]                        # (B, 1536) -- CLS token alone

Readout: CLS token alone. h[:, 0]

Embedding dimension: 1536

Token layout: num_prefix_tokens = 5 (1 CLS + 4 register tokens), then 256 spatial tokens. Patch tokens begin at index 5, not 1.

WARNINGS

1. There are TWO transforms, and which one you want depends on the benchmark

This is the one genuinely unusual inference detail in this release.

• Default (HEST, PathoROB, CPTAC and general use): Resize(256, bicubic) -> CenterCrop(224), from the checkpoint's own pretrained_cfg under timm's defaults (it carries no crop_pct and no interpolation).
• THUNDER only: Resize((224, 224)) -- a squash resize with no crop. THUNDER's get_openmidnight hand-builds this transform.

Using the generic resize+crop path on THUNDER puts the base model roughly 6.2 F1 below the published leaderboard on segmentation, and about a point below on kNN and linear probing. Match the transform to the benchmark you are reproducing.

2. Normalisation is ImageNet, unlike kaiko-ai/midnight

OpenMidnight replicates Midnight's recipe but uses ImageNet statistics (mean = (0.485, 0.456, 0.406), std = (0.229, 0.224, 0.225)), whereas kaiko-ai/midnight demands (0.5, 0.5, 0.5) / (0.5, 0.5, 0.5). The two are not interchangeable. The original training checkpoint stores no normalisation at all; these statistics come from SophontAI's own model card.

3. Readout is CLS alone, and four register tokens sit in front of the patches

num_prefix_tokens = 5. Patch/dense tokens begin at index 5.

Un-annealed checkpoints

The 1-SE selection rule picked steps inside the 200-step warmup, so these weights come from a mid-warmup, un-annealed point of the schedule -- not from the end of a completed LR decay. That is what the selection rule chose and what the paper reports, but it is unusual enough to state plainly.

Results

Base model versus these merged weights. Values are read from the SPECTRA paper's tables. n = 3 seeds; the interval is mean +/- 2SD across the three seeds.

Metric	Base model	SPECTRA (merged, n=3 seeds, mean +/- 2SD)
PathoROB mean robustness index (cross-centre)	0.618	0.879 +/- 0.007
PLISM top-1 retrieval across scanners	0.703	0.948 +/- 0.015
PLISM top-1 retrieval across stains	0.486	0.782 +/- 0.029
HEST mean Pearson r	0.3902	0.4048 +/- 0.0014
CPTAC AUC	0.6561	0.6844 +/- 0.0022

Training

Base-model weights	modified -- the LoRA delta has been folded in
Method	LoRA (PEFT), r = 32, lora_alpha = 64 (scaling alpha/r = 2.0), lora_dropout = 0.0, bias = none, no DoRA, no rsLoRA
Optimiser	AdamW, lr 1e-4, weight decay 0.05, grad clip 1.0
Schedule	500 steps total, 200 warmup
Objective	InfoNCE over registered PLISM tiles, split CLS / mean heads (weights 0.5 / 0.5), temperature 0.07
Batch geometry	4 groups x 64 tiles, 2 acquisition conditions per batch, 900-tile grid sampler
Same-core masking	on (same_core_logit_bias_cls = +3.0, same_core_logit_bias_mean = -inf)
Checkpoint selection	1-SE rule on the PathoROB robustness-index curve, applied per seed
Precision	bf16 autocast during training; the released weights are fp32

Checkpoints were written every 50 steps; only the 1-SE-selected step per seed is released. Because selection is per seed, the three seeds of a backbone need not sit at the same step -- and because the selected steps fall inside the 200-step warmup, the released checkpoints are un-annealed. This is deliberate (the 1-SE rule picked them) but it means they are not the end of a completed LR schedule.

Full run hyper-parameters are in each seed folder's training_config.json.

training_config.json encoding note. The training code writes same_core_logit_bias_mean as negative infinity, and Python's json module emits the bare token -Infinity. That token is not valid JSON and is rejected by JSON.parse and most non-Python parsers. In the released files it is encoded as the string "-Infinity" so the file parses everywhere. Read it back with float("-inf"). That is the only edit made to the training configuration.

What is NOT in this repository

Nothing else from the training run is needed to reproduce the published numbers. The contrastive projector heads (projector*.pt, projector_heads.json) and the GeM pooling head (pool_head.pt) are training-only machinery and are deliberately not released. In particular the GeM pooling head is vacuous at inference: with it applied the features are bit-identical to the plain readout below.

Citation

@inproceedings{spectra2026,
  title     = {SPECTRA: cross-acquisition robustness for pathology foundation models},
  author    = {TODO: author list},
  year      = {2026},
  note      = {TODO: confirm venue and year before citing -- submitted to NeurIPS 2026},
  url       = {https://github.com/TODO-org/spectra}
}

Code: SPECTRA on GitHub (TODO: fill in the canonical repository URL before publishing).

Merge verification

Every seed in this repository was verified at staging time, on CPU in fp32:

Folder	Modules merged	merged-vs-base feature diff (mean / max abs)	Reload bit-identical
seed0/	160	0.1175 / 25.8296	yes
seed1/	160	0.1322 / 26.8647	yes
seed2/	160	0.1186 / 23.0432	yes

"Reload bit-identical" means: the merged model was serialised to model.safetensors, re-instantiated from the architecture, re-loaded with strict=True, and its features on a fixed input compared with torch.equal against the in-memory merged model. All differences were exactly zero.

The merged-vs-base column proves the merge is not a no-op: every one of the merged modules had a non-zero delta norm, and the resulting features differ substantially from the base model's on the same input.

Licence

See LICENSE (and NOTICE, where present) in this repository. These are merged full weights, i.e. a derivative of the base model, so the base model's licence governs this artifact -- it is not merely the SPECTRA delta's licence.