File size: 2,330 Bytes
e749f92 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 | ---
license: cc0-1.0
language:
- hi
- en
tags:
- hinglish
- text-generation
- from-scratch
pipeline_tag: text-generation
---
# Chad
A small from scratch Hinglish chat model, around 100M params. Built and trained end to end: a byte level BPE tokenizer, a Llama style decoder (RoPE, grouped query attention, SwiGLU, RMSNorm), pretrained on romanized Hinglish, then fine tuned to chat in a chill Gen Z voice.
This repo holds every version in one place so the lineage is easy to follow.
## The lineage
The base model is pretrained once. Everything after that is a fine tune on top of it.
- **base (v22)**: the pretrained model, around 4B tokens of romanized Hinglish. Knows the language but not how to chat. Val loss about 3.77.
- **sft-v2**: the first good chat version, pure distillation. Clean and on character. This is the champion of the SFT runs.
- **sft-v1**: an earlier mining attempt that came out wholesome instead of funny. Kept for the record.
- **sft-v3**: v2 plus a big general Hinglish set. Broader, but the voice got diluted.
- **sft-v4 / v4.2 / v4.3**: distill heavy blends with a calculator tool and gif tags added. v4 over roasts, v4.2 dials it back, v4.3 is the widest blend.
- **dpo-final**: preference tuned on top of SFT so it leans toward the sharper reply on its own. The latest stage.
## What is in here
- `base/v22.pt`: the pretrained base (PyTorch checkpoint).
- `checkpoints/`: the raw trained weights, one per version. These are the source of truth.
- `transformers/`: the same models exported to Hugging Face format (load with `from_pretrained`). These are what the browser app runs after quantization.
- `experiments/`: side runs and A/B arms, kept so nothing is lost (the v1 continued branches, older DPO iterations).
## How to load one
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("vermarjun/chad", subfolder="transformers/v2")
t = AutoTokenizer.from_pretrained("vermarjun/chad", subfolder="transformers/v2")
```
The architecture is plain LlamaForCausalLM (hidden 768, 12 layers, 12 query heads, 3 kv heads, vocab 32000, context 1024).
The raw `.pt` files use the project's own model class, not transformers. Use the `transformers/` exports if you just want to run the model.
No private chat data was used at any stage.
|