Nerthus-Project/OE_Lemmatiser_Retrained

Old English Lemmatiser — Nerthus resource-enhanced release

A character-level encoder--decoder Transformer that jointly predicts the lemma and the UPOS tag of an Old English word form.

This is a new model, not an update of Nerthus-Project/Old_English_Lemmatiser. That repository remains the published baseline of Martin Arista et al. (2026) and is the model to cite for the figures in the paper. This release retrains the same architecture on repaired data and adds a lexicon-constrained decoder. The two models emit lemmas in different notations and their accuracy figures are not directly comparable; see Results below.

The gain comes from two places, neither of which is a change to the architecture:

1. a soft lexicon prior over the dictionary's headword list, applied when re-ranking a 10-best beam at inference time;
2. training on a repaired gold standard, produced in seven passes.

The repair is where most of the gain is, and it was done in this order:

• Format. Lemmas that were not headwords at all — glossed dictionary entries such as gnorn ‘sad’, paradigm strings such as se-seo-þæt — reduced to the headword.
• Notation. Case folded, macrons removed, eth unified with thorn, so that Æðelfrið, æðelfrið, æþelfrið and æþelfriþ are one string.
• Dictionary adjudication. Where the corpus gave one form two lemmas in different places, the dictionary settled it whenever it could do so unambiguously.
• The prefix rule. Where the competing lemmas differ only by ge-, the form itself decides: geladiege is geladian, not ladian. This consults no external resource and so introduces no circularity.
• Hand adjudication. The 657 cases no rule could settle were decided by a specialist. Cases where a form's tokens genuinely belong to different lexemes were not forced to one answer; they are left as the annotators wrote them.
• Canonicalisation. Residual editorial apparatus resolved, and a curated list of orthographic doublets unified, so that weorod and werod are one headword.

Lemmas are emitted in that harmonised convention, which is a deliberate choice: a lemmatiser that reproduced the corpus's inconsistencies would be less useful downstream than one that does not.

Usage

from lemmatizer import OldEnglishLemmatizer

lem = OldEnglishLemmatizer.from_pretrained(".")

lem.lemmatize("cyninges")
# ('cyning', 'NOUN')

lem.lemmatize_batch(["cwæð", "godan", "þurh"])
# [('cweþan', 'VERB'), ('god', 'ADJ'), ('þurh', 'ADP')]

lem.lemmatize_text("se cyning cwæð to his folce")
# [{'word': 'se', 'lemma': ..., 'pos': ...}, ...]

Beam size and the strength of the prior are read from config.json and can be overridden per call:

lem.lemmatize_batch(words, lexicon_bonus=0)   # plain beam search, no prior
lem.lemmatize_batch(words, beam_size=1, lexicon_bonus=0)   # greedy, as published

Input and output notation

Input forms are lowercased and stripped of macrons before encoding, so gōdan and godan behave identically. This matters: the training corpus writes no macrons, and passing one through unfolded takes the form outside the training distribution.

Output lemmas are lowercase, macron-free, and write thorn for eth. If you need the original corpus notation, you will have to map back yourself; the model does not reproduce it.

Results

condition	notation	decoding	dev	test
baseline_published	original	greedy	68.68	67.58
baseline_published	original	beam10	69.06	68.21
C0_replication	original	greedy	68.72	67.41
C1_variant	original	beam10	69.08	68.18
C2_lexicon	original	beam10	48.72	48.42
C4_nomacron	nomacron	beam10	73.19	72.22
C5_normalised	normalised	beam10	74.96	74.65
C7_variant_normalised	normalised	beam10	75.02	74.73
C6_combined_normalised	normalised	beam10	70.61	69.87
C8_conflictfree_normalised	normalised	beam10	75.02	73.96
C9_unseenonly_normalised	normalised	beam10	74.45	73.81
C10_revised	revised	beam10	77.43	77.17
C11_revised_apparatus	revised2	beam10	77.45	76.82
C12_prefix_rule	revised3	beam10	77.89	77.58
C13_adjudicated	revised4	beam10	78.41	78.45
C14_expert	revised6	beam10	78.76	78.30
C15_canonical	revised7	greedy	78.61	78.32
C15_canonical	revised7	beam10	78.81	78.51
C15_canonical (this release)	revised7	beam10 + soft_dict_4.0	81.80	81.57

Accuracy is exact string match on the lemma. POS accuracy is 82.79% on the test split and is unchanged by the prior, which improves lemmas only.

The full result set for every condition, including per-POS breakdowns and every prior strength tried, is in RESULTS.json.

On comparability. Figures in the original notation are directly comparable with the published 67.58%. Figures in the normalised and revised notations are not — they are computed against differently written target strings, and the correct reading of those blocks is the improvement over their own in-universe control, not the difference from 67.58%.

Intended use

Lemmatisation and POS tagging of Old English prose and verse, for corpus building, dictionary work and philological search. The model is trained on YCOE prose and poetry plus ParCorOE.

Limitations

Roughly 38% of the residual errors on the original gold standard trace to form types that the training corpus itself annotates inconsistently, which implies a ceiling near 87.65% for any model evaluated against it unrevised. Accuracy is markedly lower on adverbs, pronouns and prepositions than on nouns and verbs. POS accuracy is unchanged by everything in this release and sits at about 82.8%.

The lexicon prior can only help when the correct lemma is present in the inventory. Inventory coverage of unseen gold lemmas is about 86% on the held-out test split, so roughly one gold lemma in seven is out of reach of the prior by construction.

The prior is tuned to maximise corpus-level accuracy and will occasionally override a correct prediction. On the test split it changes 446 decisions, correcting 388 and spoiling 58, a ratio of about seven to one. What it spoils is now a single coherent class: tokens whose gold lemma is not an Old English headword, so that the inventory can only offer a wrong neighbour. These are embedded Latin (saluia rewritten to salfige, furore to furor, timebo to timber, silere to sel) and proper names, where the corpus convention is to leave the inflected form standing as its own lemma while the prior pulls it to the nominative (israeles to israel, daniele to daniel, tosti to tostig). Set lexicon_bonus=0 for material with heavy Latin admixture or dense onomastic content, and keep the default for running Old English prose and verse, where it is worth about three points.

The model is run in single-word (context-free) mode. It contains a sentence-encoder branch, but the released weights were selected in the context-free configuration, and lemmatize_text lemmatises each token independently rather than jointly.

The inventory has had the source dictionary's editorial apparatus removed: optional-prefix notation (cnawan(ge), expanded to both cnawan and gecnawan), the hyphens marking unprefixed and bound citation forms, the & abbreviation for and-, the +e transcription of the caudate e (c+elum to cælum), palatalisation dots and breves, inline glosses, and bare numerals. Compound hyphens are handled differently: since the manuscripts never write one and the dictionary lists half its hyphenated compounds solid as well, both spellings are kept (west-seaxe and westseaxe) rather than either imposed.

The training corpus additionally has its prefix alternations resolved. Where a form type was lemmatised both geladian and ladian, the lemma's prefix is made to follow the form's, so geladiege is geladian and ladiege is ladian. This settles 437 form types that no model could previously get consistently right, and raises the ceiling on the held-out split from 90.08% to 90.94%. Numerals in particular were a live failure mode: with them in the inventory the prior rewrote dates and figures to whatever similar number happened to be a headword, turning 773 into 733 and 646 into 664.

Files

file	contents
model.pt	model weights (state dict, 26.5M parameters)
config.json	architecture, decoding and training configuration
char_vocab.json	character vocabulary (140 symbols)
lemma_inventory.json	42,531 headwords, the inventory behind the soft prior
modeling_lemmatizer.py	self-contained model definition
lemmatizer.py	loader, beam search, lexicon prior, public API
inference_example.py	runnable demo
RESULTS.json	full result set for every condition
requirements.txt	torch>=2.0.0

Training

80 epochs on 84,050 corpus rows, effective batch 72, AdamW at 1.5e-4 with cosine annealing over 4 cycles, label smoothing 0.15, POS loss weight 0.2, dropout 0.15, attested-substitution character augmentation at p=0.2, seed 42. The character encoder is initialised from a pretrained checkpoint. Best dev lemma accuracy was reached at epoch 80.

Data provenance and leakage control

Every dictionary entry and variant candidate whose surface form appears anywhere in the development or test split was discarded before training, no matter what lemma it carried. This is stricter than pair-level filtering and was adopted because the raw resources overlap the evaluation data heavily: 53.9% of test tokens have their exact gold form--lemma pair listed in the lemma dictionary, and 47.3% in the variant tables. Without the quarantine, any reported gain would have been largely memorisation.

Licence and funding

CC BY 4.0. Funded by AEI /10.13039/501100011033, grant PID2023-149762NB-100 MCIN. Nerthus Project, Universidad de La Rioja.

Citation

@article{martinarista2026lemmatiser,
  title   = {A Character-Level Transformer Lemmatiser for Old English},
  author  = {Mart{\'i}n Arista, Javier and others},
  journal = {Natural Language Processing},
  year    = {2026}
}

Please cite the Nerthus project for the lexical resources.