ndrugov/nanoVLM-encoder-free

Encoder-free nanoVLM is a minimal Vision-Language Model with no vision encoder. Images are resized (aspect-ratio preserving), patchified, and fed as raw pixel patches straight into the language model via a thin embedder + projector — the number of image ("soft") tokens varies with image size. The language backbone is a HuggingFace causal LM (Qwen/Qwen3-1.7B by default).

Usage:

This is a custom architecture, so the model is loaded with the repo's own code (not transformers directly). Clone the repo, install its dependencies, then:

from models.vision_language_model import VisionLanguageModel

model = VisionLanguageModel.from_pretrained("ndrugov/nanoVLM-encoder-free")