issai/Qolda-AVL-5B

Qolda-AVL

Qolda-AVL is a 5B audio-vision-language model designed to operate in Kazakh, Russian, and English. The model extends Qwen3-VL with an audio branch built on a fine-tuned Whisper encoder and a dedicated audio projection module. All three modalities are adapted to Kazakh through a staged training pipeline, with the audio branch covering speech recognition, speech translation, audio classification, and environmental sound captioning.

To improve audio feature injection into the language backbone, we apply the DeepStack mechanism to the audio branch, mirroring the vision processing pipeline of Qwen3-VL 💜

The model is our step towards omni-modal systems for the Kazakh language.

The name "Qolda" reflects both its design and purpose in Kazakh: "in hand" (қолда) for its compact accessibility, and "to support" (қолдау) for its assistive nature.

Evaluation Results

The model is evaluated in three languages on benchmarks covering language, vision, and audio.

Language

General Language Benchmarks: Kazakh (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	77.44	77.90	84.01
MMLU-Pro	63.39	64.86	73.54
MMLU	74.45	74.98	82.71
GPQA	43.95	48.26	59.10
ARC-easy	95.75	94.82	97.85
ARC-challenge	90.87	89.85	95.14
GSM8K	87.79	87.34	90.52
Belebele	85.88	85.20	89.22

General Language Benchmarks: English (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	83.14	84.19	88.21
MMLU-Pro	70.60	72.18	79.21
MMLU	79.41	81.52	86.52
GPQA	49.54	53.68	62.23
ARC-easy	98.15	97.43	99.03
ARC-challenge	94.11	93.94	96.50
GSM8K	94.84	94.76	96.89
Belebele	95.31	95.85	97.11

General Language Benchmarks: Russian (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	80.80	81.99	86.44
MMLU-Pro	67.17	68.66	76.84
MMLU	76.95	78.48	84.81
GPQA	46.33	50.09	59.80
ARC-easy	96.84	96.84	98.53
ARC-challenge	94.89	93.94	96.16
GSM8K	92.65	93.33	94.62
Belebele	90.07	92.58	94.33

Text-to-Text Machine Translation (xCOMET-XXL, higher is better)

Direction	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	0.7783	0.8355	0.8643
en→kk	0.6534	0.7455	0.7975
kk→en	0.7845	0.8254	0.8449
ru→kk	0.6146	0.7316	0.7813
kk→ru	0.7926	0.8452	0.8679
en→ru	0.8875	0.9156	0.9374
ru→en	0.9370	0.9495	0.9570

Kazakh-Specific Benchmarks (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	72.16	76.72	80.70
KazMMLU	69.47	74.37	79.32
KazQAD	76.19	76.11	79.84
KazCulture (PQ)	97.68	98.87	99.25
KazCulture (Q)	45.28	57.54	64.39

Vision

Vision Benchmarks: Kazakh (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	62.73	65.37	65.86
AI2D	72.49	73.85	77.49
MMStar	67.59	70.68	72.08
RealWorldQA	55.95	52.03	53.33
MathVista	67.24	70.74	72.90
OCRBench	50.39	59.55	53.51

Vision Benchmarks: English (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	75.12	77.45	80.12
AI2D	81.41	81.85	84.64
MMStar	70.62	72.71	76.30
RealWorldQA	68.63	71.63	74.90
MathVista	74.57	78.47	79.48
OCRBench	82.00	82.60	85.30

Vision Benchmarks: Russian (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	56.52	59.97	63.42
MMStar	67.23	70.28	73.73
RealWorldQA	64.18	61.70	67.84
OCRBench	38.14	47.93	48.69

Audio

Automatic Speech Recognition (WER, lower is better)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Kazakh — mean (unnorm)	0.2027	0.1703	0.1684
Kazakh — mean (norm)	0.1874	0.1567	0.1551
Kazakh — cleaned (unnorm)	0.1801	0.1512	0.1358
Kazakh — cleaned (norm)	0.1648	0.1359	0.1245
English — mean (unnorm)	0.1130	0.1074	0.1065
English — mean (norm)	0.0982	0.0947	0.0931
English — cleaned (unnorm)	0.0995	0.0947	0.0974
English — cleaned (norm)	0.0876	0.0844	0.0822
Russian — mean (unnorm)	0.1388	0.1403	0.1349
Russian — mean (norm)	0.1293	0.1312	0.1257
Russian — cleaned (unnorm)	0.1222	0.1230	0.1209
Russian — cleaned (norm)	0.1129	0.1142	0.1106

Speech-to-Text Translation (xCOMET-XXL, higher is better)

Direction	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	0.7349	0.7606	0.8166
kk→en	0.6924	0.7078	0.7565
kk→ru	0.6870	0.7249	0.7842
en→kk	0.6364	0.6537	0.7667
en→ru	0.8420	0.8595	0.8887
ru→kk	0.6461	0.7047	0.7755
ru→en	0.9053	0.9131	0.9279

Spoken Attribute Reasoning (SAKURA): Kazakh (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	64.02	75.12	76.42
Gender — single	81.20	83.00	86.80
Gender — multi	78.80	82.00	86.20
Animal — single	60.00	89.60	85.80
Animal — multi	52.40	72.00	75.00
Language — single	86.14	97.00	97.20
Language — multi	79.36	93.00	93.00
Emotion — single	36.47	46.40	48.20
Emotion — multi	37.80	38.00	39.20

Spoken Attribute Reasoning (SAKURA): English (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	68.04	79.12	79.15
Gender — single	88.80	87.00	89.40
Gender — multi	84.60	88.60	88.20
Animal — single	57.20	89.00	87.00
Animal — multi	69.40	86.80	85.00
Language — single	86.75	97.20	97.60
Language — multi	88.20	95.60	95.40
Emotion — single	34.80	47.40	49.80
Emotion — multi	34.60	41.40	40.80

Spoken Attribute Reasoning (SAKURA): Russian (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	66.34	77.60	79.42
Gender — single	82.80	85.20	91.00
Gender — multi	80.80	84.80	87.60
Animal — single	60.32	87.40	88.80
Animal — multi	59.40	87.80	85.80
Language — single	90.98	97.40	97.80
Language — multi	84.80	92.80	95.00
Emotion — single	35.40	45.20	47.60
Emotion — multi	36.20	40.20	41.80

Audio Captioning and Caption QA (Accuracy %, LLM-as-Judge)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Kazakh — Average	25.89	27.20	25.20
Kazakh — Audio Cap.	12.95	18.21	14.22
Kazakh — Audio Cap. QA	38.82	36.18	36.18
English — Average	26.47	34.70	32.73
English — Audio Cap.	16.76	23.01	21.04
English — Audio Cap. QA	36.18	46.38	44.41
Russian — Average	27.22	32.05	31.27
Russian — Audio Cap.	13.64	20.35	17.80
Russian — Audio Cap. QA	40.79	43.75	44.74

Spoken Mathematical QA: Kazakh (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	87.24	91.13	92.76
Short Digit	90.00	96.00	94.00
Long Digit	88.37	94.19	95.32
Single-step Reasoning	93.58	93.07	94.09
Multi-step Reasoning	77.01	81.25	87.64

Spoken Mathematical QA: English (Accuracy %)

Benchmark	Qolda-AVL-5B	Qolda-AVL-9B	Qolda-AVL-34B
Average	88.65	92.74	93.76
Short Digit	88.00	91.00	93.00
Long Digit	84.88	93.60	94.19
Single-step Reasoning	94.93	96.28	95.61
Multi-step Reasoning	86.78	90.09	92.24

Model Usage

1. Transformers inference

To run the inference with transformers, complete the preliminary setup:

uv venv venv
source venv/bin/activate
uv pip install torch accelerate transformers

Then initialize the model and processor:

import torch
from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained(
    "issai/Qolda-AVL-5B",
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto"
)

processor = AutoProcessor.from_pretrained("issai/Qolda-AVL-5B", trust_remote_code=True)

Depending on the required modalities, define the messages list:

Language:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "y = (lnx)^2 функциясының туындысын тап. JSON форматында жауап бер: {'answer': '...'}"},
        ],
    }
]

Vision-Language:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "assets/sample_image.jpg"}, # or provide link to the image
            {"type": "text", "text": "Суретті егжей-тегжейлі сипаттап бер. Неше жылқы көріп тұрсың және олардың түстері қандай?"},
        ],
    }
]

Audio-Language:

Note: The model was not trained to answer questions posed directly in the audio. Provide a detailed text instruction alongside the audio describing the task you want performed on it.

prompt = """Математикалық есепті шеш.
Respond ONLY with this JSON format: {"explanation": "<your step-by-step reasoning>", "answer": <integer or float number>}
The answer must be a number (integer or float). No text, no units, just the number.
"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "assets/sample_audio.wav"}, # or provide link to the audio
            {"type": "text", "text": prompt}
        ],
    }
]

Audio-Vision-Language:

messages = [
    {
        "role": "user",
        "content": [
            {"type": "audio", "audio": "assets/question_audio.wav"},
            {"type": "image", "image": "assets/sample_image.jpg"},
            {"type": "text", "text": "Answer the question"},
        ],
    }
]

Finally, pass the messages to the model for inference:

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

generated_ids = model.generate(
    **inputs,
    max_new_tokens=4096,
    temperature=0.7,
    top_p=0.95,
    top_k=20,
    do_sample=True,
    repetition_penalty=1.0,
)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True)

print(processor.batch_decode(generated_ids_trimmed, skip_special_tokens=True)[0])

2. vLLM inference

Alternatively, you can run the model via a vLLM server. Note that we use a custom vLLM package. First, complete the preliminary setup:

uv venv venv
source venv/bin/activate

# Install this fork (precompiled binaries)
git clone https://github.com/IS2AI/vLLM-Qolda-AVL.git
cd vLLM-Qolda-AVL
VLLM_USE_PRECOMPILED=1 uv pip install -e .

Then start the OpenAI-compatible server (adjust parameters to your settings):

vllm serve issai/Qolda-AVL-5B \
    --served-model-name qolda-avl \
    --trust-remote-code \
    --tensor-parallel-size 4 \
    --dtype bfloat16 \
    --max-model-len 16384 \
    --limit-mm-per-prompt '{"audio": 1, "image": 1}'

To run inference, you can use the following code:

import base64
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1", 
    api_key="EMPTY"
)

def encode_audio_base64(path: str | Path) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

def encode_image_base64(path: str | Path) -> str:
    with open(path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")

audio_path = "assets/sample_audio.wav"
audio_b64 = encode_audio_base64(audio_path)

stream = client.chat.completions.create(
    model=client.models.list().data[0].id,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "input_audio",
                    "input_audio": {
                        "data": audio_b64,
                        "format": "wav",
                    },
                },
                {
                    "type": "text",
                    "text": (
                        "Analyze the voice in the audio and identify the speaker's "
                        "gender (male or female). Also transcribe what is said. "
                        "Return your answer as JSON in the following format: "
                        '{"answer": "<male or female>",'
                        '"transcription": "<transcription>"}'
                    ),
                },
            ],
        }
    ],
    max_tokens=4096,
    temperature=0.7,
    top_p=0.8,
    stream=True,
    stream_options={"include_usage": True},
)

text = ""
usage = None
for chunk in stream:
    if chunk.usage:
        usage = chunk.usage
    if chunk.choices and chunk.choices[0].delta.content:
        token = chunk.choices[0].delta.content
        print(token, end="", flush=True)
        text += token

License

Apache License 2.0

Citation

@article{qolda-avl-bdcc,
AUTHOR = {Arystanbekov, Batyr and Maxutov, Akylbek and Nurimanov, Aspandiyar and Varol, Huseyin Atakan},
TITLE = {Extending a Vision–Language Model with Audio Understanding: Introducing Qolda-AVL for the Kazakh Language},
JOURNAL = {Big Data and Cognitive Computing},
VOLUME = {10},
YEAR = {2026},
NUMBER = {6},
ARTICLE-NUMBER = {192},
URL = {https://www.mdpi.com/2504-2289/10/6/192},
ISSN = {2504-2289},
DOI = {10.3390/bdcc10060192}
}