Latent-VC-9B

Latent Video Cache for Video Reasoning

Introduction

Latent-VC (Latent Visual Cache) introduces a recurrent latent visual cache inside the decoder of a large multimodal model to mitigate Visual Anchoring Decay in long-form video reasoning. Instead of relying on a pure read-once, generate-many pipeline, Latent-VC constructs a compact latent visual memory before answer generation, enabling the model to preserve grounding to visual evidence throughout reasoning.

This repository hosts the Latent-VC-9B model, built on Qwen3.5-9B-Base and trained with two stages: Supervised Fine-Tuning (SFT) with contrastive cache alignment, followed by GRPO with vision-grounded rewards and latent grounding supervision.

Model Details

• Base model: Qwen3.5-9B-Base
• Training stages: SFT → GRPO
• Task: Grounded long-form video reasoning
• Architecture innovation: Recurrent latent visual cache inserted into the decoder

Usage

1. Install dependencies

pip install -U transformers torch accelerate opencv-python pillow numpy tqdm huggingface_hub

2. Minimal inference example

import cv2
import numpy as np
from PIL import Image
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_PATH = "BRZ911/Latent-VC-9B"


def extract_frames(video_path, max_frames=64):
    """Extract frames from a video.
    If the video is shorter than `max_frames` seconds, sample at 1 FPS;
    otherwise, sample `max_frames` frames evenly across the whole video.
    """
    cap = cv2.VideoCapture(video_path)
    fps = cap.get(cv2.CAP_PROP_FPS)
    total_frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
    duration = total_frames / fps
    frame_indices = (
        range(0, total_frames, max(int(fps), 1))
        if duration < max_frames
        else np.linspace(0, total_frames - 1, max_frames, dtype=int)
    )
    frames = []
    for idx in frame_indices:
        cap.set(cv2.CAP_PROP_POS_FRAMES, int(idx))
        ok, frame = cap.read()
        if ok:
            frames.append(Image.fromarray(cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)))
    cap.release()
    return frames


model = AutoModelForImageTextToText.from_pretrained(
    MODEL_PATH, dtype=torch.bfloat16, device_map="auto"
)
processor = AutoProcessor.from_pretrained(MODEL_PATH)

frames = extract_frames("path/to/your_video.mp4", max_frames=64)
question = (
    "What is happening in the video?\n"
    "First think step-by-step in <think> tags, then give your final answer "
    "in <answer> tags."
)

messages = [{
    "role": "user",
    "content": [{"type": "image", "image": f} for f in frames]
    + [{"type": "text", "text": question}],
}]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=1024, do_sample=False)

response = processor.batch_decode(
    output_ids[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0]
print(response)

The model is not gated, so no access token is required to download it. If you hit rate limits, you can authenticate with huggingface-cli login or by setting the HF_TOKEN environment variable.

3. Batch benchmark evaluation

This repo also ships eval_all_benchmarks.py (and a convenience wrapper eval_all_benchmarks.sh) to evaluate the model on full video benchmarks (VideoMME, MVBench, TempCompass, VideoMMMU, VSI-Bench, MMVU):

huggingface-cli download BRZ911/Latent-VC-9B \
    eval_all_benchmarks.py eval_all_benchmarks.sh --local-dir .

python eval_all_benchmarks.py \
    --model_path BRZ911/Latent-VC-9B \
    --file_name latent_vc_9b \
    --eval_dir ./Evaluation \
    --output_dir ./eval_results \
    --datasets videomme,mvbench,tempcompass,videommmu,vsibench,mmvu \
    --max_frames 64 \
    --max_new_tokens 1024

--eval_dir should point to a local copy of the benchmark data (e.g. from Video-R1/Video-R1-eval), each expected as <eval_dir>/eval_<dataset_name>.json plus the referenced video files.

For full installation, training, and evaluation instructions, see the official GitHub repository.

Training Data

The model is trained on the Latent-VC-Data dataset.

Performance

Latent-VC consistently outperforms strong CoT and SFT+GRPO baselines across six video benchmarks, especially on grounding-intensive and long-video tasks, while achieving higher accuracy with substantially shorter responses.

License

Please refer to the GitHub repository for license details.

Citation

If you find this work useful, please cite:

@misc{zhang2026latentvisualcachevideo,
      title={Latent Visual Cache for Video Reasoning},
      author={Yongheng Zhang and Zhipeng Xu and Hao Wu and Yinghui Li and Di Yin and Xing Sun and Philip S. Yu},
      year={2026},
      eprint={2607.02607},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.02607},
}