ag-news-classifier

A DeBERTa-v3-base model fine-tuned to classify English news text into three topics: Sports, Business, and Sci/Tech.

This is a 3-class model, not the standard 4-class AG News setup. The World category was removed from both the training and evaluation splits, and the remaining labels were remapped to 0, 1, 2.

Labels

id	label
0	Sports
1	Business
2	Sci/Tech

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Bubunur/ag-news-classifier")
clf("The stock market rallied after the central bank's decision")

Or directly:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "Bubunur/ag-news-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

labels = ["Sports", "Business", "Sci/Tech"]
text = "Researchers unveiled a chip that runs models on-device"

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]

print(labels[probs.argmax()], probs.max().item())

Training data

AG News, filtered to remove the World class.

split	examples
train	90,000
test	5,700

Classes are perfectly balanced: 30,000 training and 1,900 test examples each. Each example is a news title concatenated with a short description. Texts are short — truncation at 128 tokens affects almost no examples.

Training procedure

Fine-tuned from microsoft/deberta-v3-base using the HuggingFace Trainer.

hyperparameter	value
epochs	3
batch size	16
learning rate	2e-5
optimizer	AdamW
weight decay	0.01
LR schedule	linear with 6% warmup
max sequence length	128 (dynamic padding)
precision	fp16 mixed precision
seed	42
hardware	single GPU (Google Colab)
wall-clock time	~50 min

Per-epoch results:

epoch	training loss	validation loss	accuracy	macro F1
1	0.1439	0.1429	0.9542	0.9542
2	0.1645	0.1405	0.9563	0.9563
3	0.1045	0.1745	0.9574	0.9573

The epoch-3 checkpoint was selected on accuracy. Note that validation loss rose sharply at epoch 3 while accuracy continued to climb — a typical sign that the model is becoming overconfident on examples it already gets right. The accuracy gain over epoch 2 (+0.11 pp) is small enough to fall within run-to-run noise.

Implementation note: microsoft/deberta-v3-base ships fp16 weights. Loading it without forcing dtype=torch.float32 raises ValueError: Attempting to unscale FP16 gradients when fp16=True is enabled.

Evaluation

Measured on the 5,700-example AG News test split (World removed).

metric	value
accuracy	0.9574
macro F1	0.9573

Because the classes are perfectly balanced, accuracy and macro F1 coincide.

An earlier run of the same architecture under a slightly different configuration reached 0.9560 accuracy, with this per-class breakdown:

class	precision	recall	F1	support
Sports	0.99	0.99	0.99	1900
Business	0.95	0.92	0.94	1900
Sci/Tech	0.93	0.95	0.94	1900

              Sports  Business  Sci/Tech
Sports          1888        6        6
Business          11     1752      137
Sci/Tech          12       79     1809

The overall shape is expected to hold for the current checkpoint, but these per-class figures come from the earlier run and have not been recomputed.

Domain and period. AG News was collected from news wires in the early 2000s. Performance on social media text, long-form articles, non-news prose, or non-English text is untested and expected to be worse. Terminology that emerged after the collection period was never seen during training. The model has not been evaluated on any corpus other than AG News, so its ability to generalize to other news sources is unknown.

Inherited bias. Any bias present in microsoft/deberta-v3-base or in the AG News corpus carries over. No bias audit was performed.

Citation

@misc{zhang2015character,
  title={Character-level Convolutional Networks for Text Classification},
  author={Xiang Zhang and Junbo Zhao and Yann LeCun},
  year={2015},
  eprint={1509.01626},
  archivePrefix={arXiv}
}