imranraad/toxishield-v2

Model Card for toxic-classifier-38k

A bert-base-uncased model fine-tuned for binary toxicity classification (TOXIC / NON-TOXIC) of GitHub pull request review comments.

Model Details

Model Description

This model classifies a single GitHub pull request comment as toxic or non-toxic. It was fine-tuned from google-bert/bert-base-uncased on 38,761 labelled PR comments (the "38k detection dataset") as part of the ToxiShield project, which studies and filters toxicity in software engineering communication. The model is evaluated with stratified 10-fold cross-validation and also exported to ONNX (INT8-quantized) for lightweight/in-browser inference.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: Transformer encoder (BERT), fine-tuned for binary sequence classification
• Language(s) (NLP): English (software-engineering / code-review domain text)
• License: [More Information Needed] (base model google-bert/bert-base-uncased is released under Apache-2.0; license for the fine-tuned weights has not been set)
• Finetuned from model [optional]: google-bert/bert-base-uncased

Model Sources [optional]

• Repository: [ANONYMIZED-ORG]/toxic-classifier-38k (HuggingFace Hub, not yet published) / ToxiShield/toxicity-filter (source repo)
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Classifying individual GitHub PR/code-review comments as TOXIC or NON-TOXIC, e.g. via transformers.pipeline("text-classification", ...), to flag potentially toxic comments for human review.

Downstream Use [optional]

Integration into CI bots, code-review dashboards, or moderation tooling that triages or surfaces potentially toxic PR comments before/alongside human moderators. The ONNX INT8 export is intended for low-latency or in-browser inference in such tooling.

Out-of-Scope Use

• Not intended for languages other than English.
• Not validated on text outside the GitHub PR/code-review domain (e.g. social media, forums, chat, general web text) — the training distribution is short technical comments (median ~80 characters).
• Not intended as the sole basis for moderation, disciplinary, employment, or legal decisions — outputs should be reviewed by a human.
• Not a general-purpose toxicity/hate-speech classifier.

Bias, Risks, and Limitations

• Class imbalance: the training data is 73.9% non-toxic / 26.1% toxic (28,641 vs 10,120 of 38,761 comments), which can bias the model toward the majority (non-toxic) class.
• Recall on the toxic class is the weaker metric: across 10-fold CV, mean recall on toxic comments is 0.954 (± 0.010) vs. mean precision of 0.975 (± 0.003) — i.e. the model is somewhat more likely to miss a toxic comment than to wrongly flag a non-toxic one.
• Manual review of false positives (n=23 sampled) attributes most errors to:
  • Nuance/context misreads, e.g. sarcasm, dry humor, mockery (12/23, ~52%)
  • Technical jargon or inline code snippets read as hostile (6/23, ~26%)
  • Self-deprecating humor (2/23, ~9%)
  • General negative sentiment without toxicity (2/23, ~9%)
• The model was fine-tuned and evaluated on data from GitHub PR comments only, so its calibration will not necessarily transfer to other codebase-hosting platforms or review tools.

Recommendations

Use as a triage/assistive signal rather than an automated blocking mechanism, particularly given the lower recall on the toxic class. Expect false positives on sarcastic, self-deprecating, or jargon-heavy comments, and route model decisions through human review before any moderation action.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import pipeline

classifier = pipeline("text-classification", model="<path-or-hub-id-of-saved-model>")
classifier("this the stupidest code ever")[0]["label"]

Training Details

Training Data

38,761 labelled GitHub PR review comments (38k-detection-dataset, dataset handle [ANONYMIZED-ORG]/38k-dataset-labelled, not yet published): 28,641 NON-TOXIC (label 0) and 10,120 TOXIC (label 1). Comment length ranges from 5 to 998 characters (median ~80). Split into an 80% train / 10% test CSV for the single-run fine-tune; the full dataset is additionally used for stratified 10-fold cross-validation.

Training Procedure

Preprocessing [optional]

Comments are tokenized with the bert-base-uncased WordPiece tokenizer with truncation. The single-run fine-tune uses dynamic padding (DataCollatorWithPadding); the 10-fold cross-validation run uses fixed max_length=128 padding.

Training Hyperparameters

• Single-run fine-tune: learning rate 2e-5, per-device batch size 16, weight decay 0.01, 1 epoch, evaluate/save every epoch, best checkpoint restored at the end.
• 10-fold cross-validation run: learning rate 2e-5, per-device batch size 16, gradient accumulation 8 (effective batch size 128), weight decay 0.01, up to 20 epochs with early stopping (patience 3), stratified 10-fold split (each fold further split ~89/11 into train/validation).
• Training regime: fp16 mixed precision on GPU when available, fp32 on CPU.

Speeds, Sizes, Times [optional]

bert-base-uncased has ~110M parameters. The best checkpoint is additionally exported to ONNX and INT8-quantized for lighter-weight/in-browser inference. [More Information Needed] on wall-clock training time.

Evaluation

Testing Data, Factors & Metrics

Testing Data

Stratified 10-fold cross-validation over the full 38,761-sample dataset (per-fold results in results/kfold-metrics/cross_validation_results.csv); the single-run fine-tune is additionally evaluated on the held-out 10% test split.

Factors

No subpopulation disaggregation performed; results are reported per cross-validation fold and averaged.

Metrics

Accuracy, precision, recall, and F1 (binary, positive class = TOXIC), chosen to capture both overall correctness and toxic-class-specific performance given the class imbalance.

Results

10-fold cross-validation (mean ± std over 10 folds):

Metric	Mean	Std
Accuracy	0.9818	0.0023
Precision (toxic)	0.9753	0.0033
Recall (toxic)	0.9543	0.0096
F1 (toxic)	0.9647	0.0047

Baseline comparison — GPT-4o, zero-shot prompted, on the held-out test split (comparison/openai-detection-inference/):

Class	Precision	Recall	F1
Non-toxic	0.84	0.99	0.91
Toxic	0.96	0.49	0.65
Accuracy			0.86

Summary

The fine-tuned BERT model substantially outperforms zero-shot GPT-4o prompting on this task, most notably on toxic-class recall (0.95 vs. 0.49) — GPT-4o zero-shot misses roughly half of toxic comments, while the fine-tuned model catches the large majority at comparable or better precision.

Model Examination [optional]

[More Information Needed]

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: 2× NVIDIA RTX 6000 Ada Generation (49 GB), local workstation
• Hours used: [More Information Needed]
• Cloud Provider: N/A (local/on-prem)
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

bert-base-uncased (12-layer transformer encoder, 110M parameters) with a linear classification head over 2 labels (NON-TOXIC = 0, TOXIC = 1), fine-tuned with cross-entropy loss for binary sequence classification.

Compute Infrastructure

Hardware

2× NVIDIA RTX 6000 Ada Generation (49 GB each).

Software

Python 3.11, PyTorch 2.5.1 (CUDA 12.1 build), Transformers 4.57.6, Datasets 5.0.0, 🤗 Evaluate, scikit-learn, Optimum/ONNX Runtime (for INT8 export).

Citation [optional]

BibTeX:

[More Information Needed]

APA:

[More Information Needed]

Glossary [optional]

• PR: Pull request (GitHub code-review unit).
• TOXIC / NON-TOXIC: The two output labels (id 1 / id 0 respectively).

More Information [optional]

[More Information Needed]

Model Card Authors [optional]

[More Information Needed]

Model Card Contact

[More Information Needed]