README.md

---
license: mit
language:
- az
library_name: aztext
tags:
- azerbaijani
- nlp
- text-processing
- deasciify
- tokenizer
- turkic
- low-resource
---

# aztext

Lightweight, **dependency-free** Azerbaijani text-processing toolkit — the small utilities every
Azerbaijani NLP project re-implements, done once and done correctly. Pure Python standard library, no
`numpy`/`torch`/`regex` required.

Built as part of an open Azerbaijani LLM stack (tokenizer → dataset → model → evals).

## Why

Azerbaijani (Latin script) adds `ə ğ ı i ö ş ç ü` beyond ASCII, and the **dotted/dotless i** distinction
(`i`≠`ı`, `İ`≠`I`) is meaningful — but Python's `str.lower()/upper()` get it wrong, people type without
diacritics, and Azerbaijani is easily confused with Turkish. `aztext` handles these correctly.

## Install

```bash
pip install -e aztext          # from this repo
```

## Usage

```python
import aztext

# Correct Turkic-i-aware casing (Python's str.upper gets this wrong)
aztext.az_upper("işıq")            # -> "İŞIQ"
aztext.az_lower("İSTİQLAL")        # -> "istiqlal"

# Restore diacritics to ASCII-typed text (best-effort, dictionary-based)
aztext.deasciify("ucun cixis")     # -> "üçün çıxış"
aztext.ascii_fold("gözəl çıxış")   # -> "gozel cixis"

# Azerbaijani vs Turkish language ID (heuristic; ə is the key signal)
aztext.is_azerbaijani("Mən kitab oxuyuram.")             # -> True
aztext.detect_language("Ben kitap okuyorum.")            # -> ("tr", 0.87)

# Numbers to Azerbaijani words
aztext.num_to_words(1234)          # -> "min iki yüz otuz dörd"
aztext.num_to_words(-5)            # -> "mənfi beş"

# Normalization, tokenization, script detection
aztext.normalize("Gözəl   şəhər — “Bakı”.")   # NFC, quotes/dashes, whitespace
aztext.word_tokenize("Bakı, paytaxtdır.")      # -> ["Bakı", "paytaxtdır"]
aztext.sent_tokenize("Bir. İki! Üç?")          # -> ["Bir.", "İki!", "Üç?"]
aztext.is_latin_azerbaijani("Azərbaycan dili") # -> True
```

## API

| function | does |
|---|---|
| `normalize(text)` | NFC, mojibake repair, zero-width strip, quote/dash + whitespace cleanup (preserves casing & Az letters; idempotent) |
| `deasciify(text)` | restore Azerbaijani diacritics on ASCII-typed text (dictionary best-effort) |
| `ascii_fold(text)` | strip diacritics (`ə→e`, `ş→s`, …) |
| `is_azerbaijani(text)` / `detect_language(text)` | Az-vs-Tr-vs-other heuristic ID |
| `num_to_words(n)` | Azerbaijani cardinal spelling (0 … < 10¹², negatives) |
| `word_tokenize` / `sent_tokenize` | explicit-alphabet word/sentence tokenizers |
| `is_latin_azerbaijani` / `script_ratios` | Latin-vs-Cyrillic/Arabic script detection |
| `az_lower` / `az_upper` | Turkic-i-aware case mapping |

## Limitations (honest)

- **`deasciify` is dictionary-based best-effort.** It restores common words; unknown words pass through
  unchanged, and genuinely ambiguous folds (e.g. `el` → *el* "people" vs *əl* "hand") resolve to a single
  listed form. It is not a language model.
- **`detect_language` is a lightweight heuristic**, not a trained classifier — tuned for the Az/Tr split.
  An optional `fasttext` model is used as a tiebreaker only if `AZTEXT_FASTTEXT_MODEL` points to one.
- Latin-script Modern (North) Azerbaijani only; Cyrillic/Perso-Arabic are detected but not transliterated.

## Tests

```bash
python -m pytest aztext/tests -q      # or, dependency-free:
python aztext/tests/test_aztext.py
```

## License

MIT.