🌟 BalBERT (shahbakhsh/BalBERT)

The Foundational Research-Grade Transformer Encoder for Balochi NLP

---

📌 Model Overview

BalBERT is the first research-grade Transformer encoder domain-adapted specifically for the Balochi language (bal). Built via continuous Domain-Adaptive Pretraining (DAPT) of xlm-roberta-base on a normalized monolingual corpus of 97,624 Balochi sentences, BalBERT provides a domain-adapted foundation model for all downstream Balochi natural language processing tasks.

• Author / Developer: Shah Bakhsh (@shahbakhsh)
• Model Type: Masked Language Model (Transformer Encoder)
• Language: Balochi (bal)
• Base Architecture: xlm-roberta-base
• License: Apache-2.0
• Repository: github.com/shah-bakhsh/BalBERT

---

🚀 Benchmark & Performance Highlights

Metric	xlm-roberta-base (Baseline)	BalBERT (Ours)	Improvement
MLM Loss	5.8100	1.7681	-69.6% Reduction
Perplexity	333.63	7.40 (Val: 5.86)	+97.8% Drop
Masked Prediction Quality	Generic Punctuation (،)	Contextual Balochi Words (اے, آئی)	Linguistically Coherent
Embedding Affinity	Collapsed uniform score	High dynamic margin	Enhanced Representations

---

💻 Quick Start & Code Examples

1. Fill-Mask Pipeline (Inference)

You can run token completion directly using the Hugging Face pipeline:

from transformers import pipeline

# Load pipeline directly from Hugging Face Hub
unmasker = pipeline("fill-mask", model="shahbakhsh/BalBERT")

# Predict masked word in Balochi
text = "اے دوئیں دبستانانی <mask> دیگءَ پرکے ھست۔"
results = unmasker(text)

print("Top Predictions:")
for res in results[:5]:
    print(f"Token: {res['token_str']:<15} | Score: {res['score']:.4f}")

---

2. Direct PyTorch Model & Tokenizer API

import torch
from transformers import AutoTokenizer, AutoModelForMaskedLM

MODEL_ID = "shahbakhsh/BalBERT"

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForMaskedLM.from_pretrained(MODEL_ID)

text = "پاکستان ءِ ردءَ اے <mask> باز ارزښت دار اِنت۔"
inputs = tokenizer(text, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits

mask_token_index = torch.where(inputs["input_ids"] == tokenizer.mask_token_id)[1]
mask_logits = logits[0, mask_token_index, :]
top_5_tokens = torch.topk(mask_logits, 5, dim=1).indices[0].tolist()

print("Top Mask Completions:")
for token_id in top_5_tokens:
    print(f" • {tokenizer.decode([token_id])}")

---

3. Extracting Contextual Sentence Embeddings

import torch
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("shahbakhsh/BalBERT")
model = AutoModel.from_pretrained("shahbakhsh/BalBERT")

sentence = "بلۆچی زبان ءِ شَرّێن زانتکار"
inputs = tokenizer(sentence, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    # Tensor Shape: [batch_size, sequence_length, 768]
    last_hidden_state = outputs.last_hidden_state

print("Embedding Shape:", last_hidden_state.shape)

---

📊 Evaluation & Qualitative Analysis

Masked Word Prediction Comparison

Given the Balochi sentence:

اے جستءِ پسو گپءُ درجنی دگہ جنگے رودین ایت کہ ادا <mask> ءِ سرجمیں پہناتانی گیشواری المی نہ اِنت۔

xlm-roberta-base Top Predictions:
1. ، (4.5%) 
2. ِ (3.0%) 
3. کی (2.9%) 
4.   (2.3%) 
5. ، (2.0%) 

BalBERT Top Predictions (Ours):
1. اے     (28.4%) [Demonstrative: "this"]
2. آئی    (28.3%) [Pronoun: "its/his/her"]
3. انسانی (13.1%) [Adjective: "human"]
4. پہ     (2.5%)  [Preposition: "for"]
5. کہ     (1.6%)  [Conjunction: "that"]

---

⚙️ Model Specifications & Training Configuration

Architecture Specs

• Base Model: XLM-RoBERTa Encoder (xlm-roberta-base)
• Total Parameters: 278,295,186 (~278M)
• Hidden Layers: 12
• Attention Heads: 12
• Hidden Dimension: 768
• Vocab Size: 250,002
• Max Sequence Length: 128 tokens

Pretraining Hyperparameters

• Objective: Masked Language Modeling ($p=0.15$ dynamic masking)
• Learning Rate: 5e-5 (Linear decay)
• Warmup Ratio: 0.06
• Weight Decay: 0.05
• Per-Device Batch Size: 16 (Gradient Accumulation: 2 -> Effective Batch Size: 32)
• Precision: bf16 Mixed Precision
• Hardware: NVIDIA Tesla T4 GPU (16 GB VRAM) under CUDA 12.8

Dataset Overview

• Source File: sentences_clean.txt
• Cleaned Sentences: 97,624 sentences
• Train / Val Split: 95% Train (92,742 sentences) / 5% Validation (4,882 sentences)
• Mean Length: 91.3 chars / 19.8 words per sentence
• Seed: 42

---

🗺 Balochi NLP Ecosystem Roadmap

BalBERT acts as the pre-trained feature extractor for the entire Balochi NLP stack:

$$\text{BalCorpus}⟶\text{BalTokenizer}⟶\mathbf{B}\mathbf{a}\mathbf{l}\mathbf{B}\mathbf{E}\mathbf{R}\mathbf{T}⟶\text{BalPOS v2}⟶\text{BalNER}⟶\text{BalMorph}⟶\text{BalParser}\backslash text\{BalCorpus\}\; \backslash longrightarrow\; \backslash text\{BalTokenizer\}\; \backslash longrightarrow\; \backslash mathbf\{BalBERT\}\; \backslash longrightarrow\; \backslash text\{BalPOS\; v2\}\; \backslash longrightarrow\; \backslash text\{BalNER\}\; \backslash longrightarrow\; \backslash text\{BalMorph\}\; \backslash longrightarrow\; \backslash text\{BalParser\}$$

• BalCorpus: Curated large-scale monolingual Balochi text dataset.
• BalTokenizer: Native Balochi BPE tokenizer.
• BalBERT: Domain-adapted transformer backbone encoder.
• BalPOS v2 (Next Release): Universal Part-of-Speech tagger fine-tuned on shahbakhsh/BalBERT.
• BalNER: Named Entity Recognition model for Balochi.
• BalMorph: Morphological analyzer.
• BalParser: Dependency parser based on Universal Dependencies (UD).

---

⚠️ Intended Use & Limitations

Intended Use

• Fine-tuning on downstream Balochi token classification (POS, NER) and text classification tasks.
• Extracting contextual token and sentence embeddings for Balochi NLP research.

Limitations

• Encoder-Only: BalBERT is a 278M parameter encoder model and cannot generate long-form text.
• Text Domain: Trained primarily on written Balochi text; oral dialects or non-standard spelling variants may require additional fine-tuning.

---

📑 Citation

@misc{shahbakhsh2026balbert,
  author       = {Shah Bakhsh},
  title        = {BalBERT: Domain-Adaptive Pretraining of XLM-RoBERTa for Balochi Natural Language Processing},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/shahbakhsh/BalBERT}},
  note         = {GitHub Repository: shah-bakhsh/BalBERT}
}

---

📬 Author & Contact

• Author: Shah Bakhsh
• Hugging Face: @shahbakhsh
• GitHub: @shah-bakhsh
• Repository: github.com/shah-bakhsh/BalBERT