What is OpenLLaVA?

OpenLLaVA is a comprehensive open-source framework for injecting vision capabilities into any language model. It provides a complete pipeline — from model construction through training, inference, serving, export, and evaluation — all accessible through a unified Python API and CLI.

The framework supports any LLM architecture (Llama, Mistral, Qwen, Gemma, Phi, DeepSeek, and more) and any HuggingFace-compatible vision encoder. It automatically detects model dimensions, constructs the appropriate projector, patches the tokenizer with visual tokens, and configures the full training and inference pipelines.

The central design goal: when a new language model drops, you should have a vision version in 48 hours.

OpenLLaVA is backend-agnostic. The same code runs on CUDA, ROCm, Apple MLX, Intel XPU, Google TPU, and CPU — with automatic hardware detection and optimal configuration selection.

Quickstart

pip install openllava        # Core
pip install openllava[cli]   # With CLI tools
pip install openllava[serve] # With serving
pip install openllava[all]   # Full installation

Inject Vision Into Any LLM

from openllava import OpenLLaVA, Backend

model = OpenLLaVA(
    llm="meta-llama/Llama-3-8B",
    vision_encoder="google/siglip2-so400m-patch14-384",
    backend=Backend.AUTO,
)

OpenLLaVA auto-detects hidden dimensions, builds the projector, and patches the tokenizer. No boilerplate. No config files.

Train with LoRA

model.lora(r=64, alpha=128, dropout=0.05)

model.train(
    phase1=dict(dataset="liuhaotian/LLaVA-Pretrain", samples=100_000),
    phase2=dict(dataset="liuhaotian/LLaVA-Instruct-150K", learning_rate=2e-4),
    resume=True,
)

model.push("my-org/my-vision-model")

FastVisionModel API

from openllava.api import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "Openllava/Yaki",
    max_seq_length=2048,
    load_in_4bit=True,
)

model = FastVisionModel.get_peft_model(model, r=16, alpha=32)

Serve as OpenAI-Compatible API

openllava serve Openllava/Yaki --port 8000

from openai import OpenAI

client = OpenAI(api_key="openllava", base_url="http://localhost:8000/v1")

response = client.chat.completions.create(
    model="yaki",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What is in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
        ],
    }],
)

Key Features

Model Construction Vision injection into any HuggingFace LLM in 3 lines AnyRes dynamic high-resolution with patch grouping YakiProjector: configurable MLP alignment Auto-detects hidden dimensions, attention heads, vocabulary size Supports LoRA-patched models Training Pipeline 3-phase training: alignment, instruction tuning, RL alignment LoRA, LoRA+, DoRA, QLoRA, Split LoRA, LoRAGA, LoRAFA BitNet ternary training (b1.58) MoE + LoRA fusion FP8 training on H100 Padding-free and sequence packing Curriculum learning RL Alignment DPO, GRPO, ORPO, PPO Composable reward functions Visual reasoning reward support

Inference and Serving Continuous batching PagedAttention (4x memory efficiency) Speculative decoding (Eagle, Medusa, NGram) KV cache: quantization, eviction, compression OpenAI-compatible FastAPI server Streaming support Optimization Suite (40+) torch.compile full-graph compilation GPTQ / AWQ / FP4 / NVFP4 quantization GaLore gradient projection torchao integration EMA training stability Selective activation checkpointing Distributed Training FSDP2, DeepSpeed ZeRO (stages 0-3) Tensor, Pipeline, Expert parallelism Ring Attention for long context Heterogeneous GPU + CPU + TPU training Auto-parallelism detection

Multi-Backend Support

Stack

Architecture

Image + Text feeds into a Vision Encoder (SigLIP2, CLIP, DINOv2, or any HuggingFace encoder), whose patch features are passed through the YakiProjector (Patch Grouping 3x3 + MLP 2-layer, mapping vision_dim x 9 to llm_dim). The projected embeddings are merged with text embeddings and passed to the Language Model (any AutoModelForCausalLM, with QLoRA 4-bit NF4 and LoRA r=64), which generates text output including <think> reasoning blocks when applicable.

Yadis Architecture

Yadis is OpenLLaVA's flagship multimodal architecture — the long-term evolution of the framework combining discrete visual tokens, MLP projection, and cross-attention per LLM layer.

# Yadis Routing — multiple vision experts with MoE router
from openllava import OpenLLaVA, experts

model = OpenLLaVA(
    llm="OpceanAI/OwO-32B",
    architecture="yadis_routing",
    experts=[
        experts.Visual("google/siglip2-so400m-patch14-384"),
        experts.OCR("deepseek-ai/DeepSeek-OCR-2"),
    ],
)

# Yadis Full — discrete tokens + cross-attention per layer
model = OpenLLaVA(
    llm="OpceanAI/OwO-32B",
    architecture="yadis_full",
    vision_encoder="google/siglip2-so400m-patch14-384",
)

OpceanAI Vision Models

OpceanAI uses OpenLLaVA to publish vision versions of new language models within 48 hours of release.

Yaki v1 Vision-language model built on Yuuki RxG 8B. Designed for complex visual reasoning with bilingual support (ES/EN). Preserves the <think> chain-of-thought behavior of the base model for multimodal tasks. Base: DeepSeek-R1-Qwen3-8B fine-tune Encoder: SigLIP 2 SO400M LoRA: r=64, alpha=128

Yaki v2 (planned) Built on Yuuki ExG 14B with cross-attention architecture (OpenLLaVA v4).

Yaki v3 (planned) Built on OwO 32B with full Yadis routing architecture, combining visual and OCR expert encoders.

Philosophy

Architecture Agnostic by Design Every existing multimodal framework is hardcoded to specific model families. OpenLLaVA is not. The projector adapts to any hidden dimension. The patcher works on any causal LM. The training engine handles any tokenizer. Speed Over Ceremony When a new model is released, the window to publish a vision version is 48 to 72 hours. OpenLLaVA is designed for that constraint — minimal configuration, automated phase management, one-command training.

Low Level Where It Matters The projector is the critical path. The CUDA kernel for the fused MLP and the C++ memory manager exist because training throughput on a single GPU is the binding constraint for a zero-budget research organization. Fully Open Apache 2.0. No gating. No commercial restrictions. The framework exists so that any researcher — with any model, any hardware, any budget — can build a competitive vision-language model.

Roadmap