SIRIS-Lab/social-sciences_binary_classifier

📗 SPECTER2 – Social Sciences Classifier (Binary Classification)

This model is a fine-tuned version of allenai/specter2_base for identifying whether a scientific publication belongs to the Social Sciences domain.

It achieves the following results on the evaluation set:

• Loss: 0.1382
• Accuracy: 0.9670
• F1 Micro: 0.9670
• F1 Macro: 0.9480
• F1 Weighted: 0.9670

Model description

This model performs binary document classification and predicts whether a publication belongs to the Social Sciences domain.

The model was trained using title and abstract text from multiple openly available datasets with native disciplinary annotations, including:

• MAG / SciDocs
• Elsevier Open Access (ASJC subject areas)
• ERC panel datasets (publications and funded projects)

Each dataset was converted into a common binary label indicating whether a document belongs to the Social Sciences according to mappings from the original classification systems. :contentReference[oaicite:0]{index=0}

Key characteristics

• Base model: allenai/specter2_base
• Task: binary document classification
• Labels:
  • False → Non-Social Sciences
  • True → Social Sciences
• Activation: softmax
• Loss: CrossEntropyLoss

Intended uses

This model is suitable for:

• Identifying Social Sciences publications
• Research information systems
• Funding portfolio analysis
• Metadata enrichment
• Bibliometric analyses

The model accepts:

• title
• abstract
• title + abstract (recommended)

Training data

Training data combines approximately 20,000 documents sampled from multiple sources:

• MAG/SciDocs
• ERC panel datasets (publications and projects)
• Elsevier Open Access publications

Each source provides its own disciplinary taxonomy. Categories corresponding to Social Sciences were mapped into a common binary classification problem. :contentReference[oaicite:1]{index=1}

Training procedure

Preprocessing

• Input text: title + abstract
• Maximum sequence length: 512 tokens
• Tokenization using the SPECTER2 tokenizer

Training hyperparameters

• learning_rate: 2e-5
• train_batch_size: 32
• eval_batch_size: 32
• num_epochs: 4
• max_length: 512
• optimizer: AdamW
• metric for best model: F1 Macro

Evaluation results

Metric	Value
Accuracy	0.9670
F1 Micro	0.9670
F1 Macro	0.9480
F1 Weighted	0.9670

Limitations

• The model predicts whether a publication belongs to the Social Sciences domain only.
• It does not distinguish between individual Social Sciences disciplines.
• Labels are derived from mappings between different disciplinary taxonomies and should be interpreted as high-level domain assignments.

Framework versions

• Transformers 4.57.1
• PyTorch 2.8.0
• Datasets 3.6.0
• Tokenizers 0.22.1