File size: 1,455 Bytes
d92c25b 455cb0d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | ---
license: mit
---
# Language Detection
A lightweight language detection tool that uses character-level n-gram features and logistic regression to identify the language of a given text.
Supported languages out of the box: English, French, German, Turkish.
Model repository: https://huggingface.co/Isa0/language-detection/
## Installation
Requires Python 3.11 or higher. Install dependencies with [uv](https://github.com/astral-sh/uv):
```bash
uv sync
```
## Usage
### Train
Train the model on the datasets in the `datasets/` directory:
```bash
uv run main.py --train
```
You can point it to a different directory with `--dir`:
```bash
uv run main.py --train --dir path/to/datasets
```
Each `.txt` file in the directory should contain one sentence per line. The filename (without extension) is used as the language label.
### Detect
Detect the language of a text string:
```bash
uv run main.py --detect "Bonjour, comment allez-vous?"
```
Output includes the predicted language and a confidence score.
## Adding Languages
Add a new `.txt` file to the `datasets/` directory named after the language (e.g. `spanish.txt`), with one sentence per line, then retrain.
## How It Works
Text is converted into character-level n-gram counts (1 to 3 characters), which capture language-specific patterns like accents, letter combinations, and suffixes. A logistic regression classifier is trained on these features and saved to disk for reuse.
|