Sounds like a good eval for my small 0.2-0.4b models

Nvm found you added it to the leaderboard

Your distractors never mention the subject. That is measurable, so I measured it.

First 1,000 items, pulled through datasets-server. 2,969 distractor segments, and exactly 0 of them contain the item's own meta.subject string. 78.3% of related segments do. So "related" and "the sentence with the topic noun in it" are very nearly the same set.

A no-model baseline says the same thing. Score each segment by the fraction of its content words that also appear in the question. No attention, no gradients, no forward pass. On 793 of 1,000 items every related segment outranks every distractor, mean margin 0.295. Only 26 items break it. The rest tie, mostly mechanism sentences that share no wording with the question.

So a Priority above 50 is reachable by token matching alone, and a model that only matches strings looks like a model that tracks causality.

The fix is cheap, because your generator already holds the pieces. meta gives you subject, driver, mechanism, outcome. Add a third segment class: same subject, same driver, no causal link. "The bread dough was left in a ceramic bowl overnight." Then above 50 means the model separated causal relevance from topic overlap, which is the thing the bench is named for.

What does Glint2 score when the distractors are lexically matched?

You said the target is attention and detail finding, not subjects. Fair. So I dropped the subject argument and ran your own scoring formula instead.

evaluation_harness.py computes priority_score = 100 * related_mean / (related_mean + unrelated_mean). I kept that and your per-item aggregation exactly, and swapped one thing: token importance becomes 1 if a token is a content word that also appears in the question, 0 otherwise. No model, no forward pass, no gradients.

All 5,000 items: priorityScore 98.88. Median 100. All 5,000 land above the ~50 your card calls no systematic preference, and 4,867 land above 90.

Flat everywhere it should not be. easy 99.09, medium 98.81, hard 98.76. By template A 98.88, B 98.72, C 98.73, D 98.73, E 99.01, F 99.20.

The cause is in the data, not the metric. Across all 5,000 items, 0 of 14,828 distractor segments contain the item's own subject string, and distractors share almost no content words with the question. So unrelated_mean sits near zero, and a ratio whose denominator is near zero saturates.

Two consequences. The null is wrong: 50 is the null for a coin flip, but the cue actually sitting in the context is lexical, so the null is about 99. And since gci is (priority + linkage) / 2, the priority half is carrying almost no information and linkage_score is doing all of the separating.

Separately, difficulty is exactly segment count. easy 4, medium 6, hard 8, zero exceptions in 5,000, and related is exactly half the segments in every single item. So hard means longer context at fixed label balance.

Credit where it is due: your label hygiene is clean. 5,000 for 5,000, no duplicate related ids, none dangling.

Caveat on my number, since it matters. Word-level tokens, not your subword tokenizer, and binary importance instead of continuous saliency, which is sharper than real attention. So read 98.88 as what lexical matching alone buys under your formula, not as a model score.

What does linkageScore alone look like across your leaderboard? If priority is pinned near 99 for everyone, that is the only axis you have left.