vcl-iisc/DynEval-Evaluator

DynEval Evaluator

DynEval Evaluator provides Qwen3-VL based vision-language evaluator checkpoints for text-to-image evaluation. The evaluator takes an image and its text prompt, generates image-specific verification questions, and scores the image using visual evidence.

The repository contains two checkpoint variants:

Variant	Subfolder	Base architecture
DynEval Evaluator 2B	DynEval-2B	Qwen3VLForConditionalGeneration
DynEval Evaluator 4B	DynEval-4B	Qwen3VLForConditionalGeneration

Both checkpoints are stored in Hugging Face transformers format.

Task Tokens

The evaluator uses task tokens internally:

<T2IA>
<IQA>
<EVALUATION>

The normal inference pipeline uses:

1. <T2IA> to extract text-to-image alignment elements from the prompt.
2. <T2IA> to generate yes/no verification questions for those elements.
3. <EVALUATION> to score the image against the generated questions on a 1--5 scale.

Installation

pip install torch transformers accelerate pillow

If the model is gated, accept access on the model page and log in:

huggingface-cli login

Loading the Model

DynEval-2B

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

repo_id = "vcl-iisc/DynEval-Evaluator"
subfolder = "DynEval-2B"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo_id,
    subfolder=subfolder,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

processor = AutoProcessor.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
)

DynEval-4B

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

repo_id = "vcl-iisc/DynEval-Evaluator"
subfolder = "DynEval-4B"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo_id,
    subfolder=subfolder,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

processor = AutoProcessor.from_pretrained(
    repo_id,
    subfolder=subfolder,
    trust_remote_code=True,
)

Recommended Inference

Use the inference script from the DynEval code repository:

CUDA_VISIBLE_DEVICES=0 python run_inference.py \
  --variant 2b \
  --prompt "a photo of a carrot" \
  --image example.jpg \
  --output-file output.json

For DynEval-4B:

CUDA_VISIBLE_DEVICES=0 python run_inference.py \
  --variant 4b \
  --prompt "a photo of a carrot" \
  --image example.jpg \
  --output-file output.json

The script prints a readable result and optionally saves JSON:

{
  "prompt": "a photo of a carrot",
  "image_path": "example.jpg",
  "elements": ["carrot (food)"],
  "questions": [
    {
      "element": "carrot (food)",
      "question": "Is there a carrot in the photo?",
      "answer": "yes"
    }
  ],
  "answers": [
    {
      "question": "Is there a carrot in the photo?",
      "score": 5
    }
  ]
}

Minimal Manual Example

The following example runs only the final <EVALUATION> step with a user-provided question. For the full DynEval pipeline, use run_inference.py.

from PIL import Image
import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

repo_id = "vcl-iisc/DynEval-Evaluator"
subfolder = "DynEval-2B"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    repo_id,
    subfolder=subfolder,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()
processor = AutoProcessor.from_pretrained(repo_id, subfolder=subfolder, trust_remote_code=True)

image = Image.open("example.jpg").convert("RGB")
questions_text = "1. Question: Is there a carrot in the photo?"

messages = [
    {
        "role": "system",
        "content": [
            {
                "type": "text",
                "text": "You are a strict visual evidence evaluator. Score based only on what you can see in the image. Return only a JSON array.",
            }
        ],
    },
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": f"<EVALUATION>\nQuestions to score:\n{questions_text}"},
        ],
    },
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True).to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)

input_width = inputs["input_ids"].shape[1]
output = processor.decode(
    generated_ids[0, input_width:],
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)
print(output)

Model Details

DynEval Evaluator 2B

• Model type: qwen3_vl
• Architecture: Qwen3VLForConditionalGeneration
• Tokenizer: Qwen2Tokenizer
• Text hidden size: 2,048
• Text layers: 28
• Attention heads: 16
• KV heads: 8
• Vision encoder depth: 24
• Vision hidden size: 1,024
• Vision patch size: 16

DynEval Evaluator 4B

• Model type: qwen3_vl
• Architecture: Qwen3VLForConditionalGeneration
• Tokenizer: Qwen2Tokenizer
• Text hidden size: 2,560
• Text layers: 36
• Attention heads: 32
• KV heads: 8
• Vision encoder depth: 24
• Vision hidden size: 1,024
• Vision patch size: 16

Intended Use

DynEval Evaluator is intended for research on multimodal and text-to-image evaluation. It can be used to compare generated images against their text prompts through dynamically generated verification questions and visual scoring.

Limitations

The evaluator is a learned model and may produce incorrect questions or scores. Use it as an automatic evaluation signal, not as a replacement for human judgment in high-stakes settings.