Daan Hoekstra PRO

Really well-timed release. The separation between task definition and runtime harness is the right design call. Anyone who has tried to reuse an existing eval under a different prompt format knows how painful it gets when those concerns are tangled together.
The per-question pairwise comparison with minimum detectable effect reported alongside the score is something more eval frameworks should do by default.
One question: how does olmo-eval handle document-understanding tasks where input is not plain text? We have been building NER pipelines for compliance documents using LayoutLMv3 and the eval loop gets tricky when ground truth alignment depends on bounding box coordinates rather than token spans. Does the task abstraction cover that, or does it fall into the ExternalEval path?