iamahmadyasin/humor-distilbert

Humor Intelligence — DistilBERT

A DistilBERT model fine-tuned to predict how funny a joke is, trained on 340k cleaned Reddit jokes from the rJokes dataset (Weller & Seppi, LREC 2020). Given a joke as input, the model outputs a single scalar: a predicted humor score on a 0–11 scale (the dataset's log-compressed community rating). This is the lightweight rung of the model ladder (66M parameters), half the size of RoBERTa-base, yet within 0.007 Spearman of it.

Results

Evaluated on a leakage-cleaned test set (see below).

Model	Params	Test Spearman	Test Pearson	Test RMSE
TF-IDF + Ridge	—	0.363	—	—
DistilBERT-128	66M	0.4118	0.4513	1.6426
RoBERTa-base-128	125M	0.4187	0.4510	1.7047
BERT-large	340M	0.423	0.463	1.657
RoBERTa-large-128	355M	0.4323	0.4701	1.6298

Paper evaluates on a test set containing ~2.4% cross-split leakage. On a comparably leaked eval, DistilBERT scores Spearman 0.4210. Despite having half the parameters of RoBERTa-base, DistilBERT achieves nearly the same ranking performance (Spearman 0.4118 vs 0.4187) and actually produces better-calibrated magnitude predictions (RMSE 1.6426 vs 1.7047).

Leakage-cleaned evaluation

The original rJokes splits contain ~2.4% of test jokes that are exact copies of training jokes (Reddit reposts). Prior work evaluated on these leaked splits. I remove the overlap and report on the clean test set (41,957 examples). I quantified the impact:

Model	Clean Spearman	Leaky Spearman	Inflation	Paper Spearman
DistilBERT (66M)	0.412	0.421	0.009	—
RoBERTa-base (125M)	0.419	0.426	0.007	—
BERT-large (340M)	0.423	0.431	0.009	0.430
RoBERTa-large (355M)	0.432	0.440	0.008	0.435

The average inflation is 0.008 ± 0.001 Spearman, consistent across four architectures of different sizes, confirming it is a dataset property and not a model-specific artifact.

Training details

• Base model: distilbert-base-uncased (66M parameters)
• Task: Single-value regression (num_labels=1, problem_type="regression")
• Dataset: rJokes, cleaned (339,499 train / 41,941 dev / 41,957 test)
• Cleaning: removed 5,707 exact duplicates, ultra-short fragments (<5 words), and ~2.4% cross-split leakage from dev/test
• Max sequence length: 128 tokens
• Epochs: 5 (best checkpoint at epoch 3 by dev Spearman)
• Effective batch size: 32 (constant across single and multi-GPU setups)
• Learning rate: 2e-5 with 6% linear warmup
• Weight decay: 0.01
• Precision: fp16
• Optimizer: AdamW (Hugging Face default)
• Seed: 42
• Hardware: Kaggle T4 ×2, ~4 hours

Label note

The rJokes score column is already log-scaled: round(ln(raw_upvotes + 1)), giving integers 0–11". It is used directly as the regression target. Do not log-transform again. This follows the paper's Section 3.1, which reduces the raw scale (0–136,353) down to integers 0–11 (the paper reports 0–10; labels of 11 are rare but present in the data)".

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

repo = "iamahmadyasin/humor-distilbert"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo)
model.eval()

joke = "I told my wife she was drawing her eyebrows too high. She looked surprised."
inputs = tokenizer(joke, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    score = model(**inputs).logits.item()
print(f"Predicted humor score: {score:.2f}")

Limitations

• Humor is subjective; the labels reflect one Reddit community's preferences, shaped by timing and virality as much as joke quality.
• The model regresses to the mean and is unreliable at the extremes of the score range (rarely predicts 0 or 6+).
• Trained on English-language Reddit jokes only.
• This is a humor ranker, not a judge of objective funniness.

Citation

Dataset:

@inproceedings{weller-seppi-2020-rjokes,
    title     = "The rJokes Dataset: a Large Scale Humor Collection",
    author    = "Weller, Orion and Seppi, Kevin",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference (LREC)",
    year      = "2020",
    pages     = "6136--6141",
    url       = "https://aclanthology.org/2020.lrec-1.753/",
}

Project

Full project: github.com/iamahmadyasin/humor-intelligence