Abstract
Multimodal large language models exhibit distinct attention patterns during generation, with attention to visual and textual modalities shifting based on semantic requirements, and these patterns can be leveraged to improve task performance through targeted interventions.
Multimodal large language models (MLLMs) generate responses autoregressively, integrating visual and linguistic information in an evolving context. Prior work on interpretability has focused on individual layers and circuits (where), leaving the token-level dynamics of multimodal computation during generation (when) underexplored. We address this gap and study attention shifts as per semantic role; tracking model attention to image, text, instruction, and previously generated tokens, One Token at a Time (OTaT). We introduce multimodal tasks that require explicit switching between visual and textual context within a single response. Across two mainstream model families and four open-weight MLLMs of varying sizes, we establish consistent patterns: attention to image peaks at tokens requiring image-derived information, instruction tokens are revisited during task transitions, and attention to previously generated tokens increases as the generation progresses. Causal attention blocking interventions validate the functional role of these trends. We profile model behavior under disrupted attention and observe responses falling back to language priors, or exhibiting cross-modal leakage, denial, or recovery. Finally, informed of the attention dynamics through our novel analysis, we propose a simple test-time intervention to boost attention to the relevant modality at the right time, significantly improving multimodal task performance.
Community
We started with a simple question: how do visual representations evolve in VLMs? ~100 linear probes later, the answer was basically 'they don't lose much.'
So we flipped the question;
From 'where' information lives, to 'when' each modality matters as the model writes its answer - One Token at a Time (OTaT).
OTaT tracks attention from the token being generated to the image, text, instruction, and everything generated so far - across the whole response, no head or layer cherry-picking.
Multiple findings emerge;
- Attention to image peaks exactly when the model writes an image-grounded word; text peaks when it does the math.
- Models re-read the instruction right at the handoff between subtasks.
- Models attend their own generated tokens as the response grows, to stay fluent.
We show that these findings are causal - by blocking each chunk at the moment it should matter:
- Block the image while it writes the fruit -> identification collapses.
- Block the instruction only at the handoff -> get partial completions.
- Block previously generated tokens → fluency breaks ('the fruit the fruit the fruit…')
More interesting discussions in the paper;
- 'The fruit in the image is Juice box C' (hallucinated fruit from the math puzzle)
- '.. not strawberry, but strawberries cut in half' (re-calibrated responses post image blocking)
Beyond Observations;
Boosting attention to the right modality at the right time makes LLaVA-OneVision jump +28.5 points on spatial reasoning - catching up to Qwen2.5-VL, with a simple test-time intervention.
Get this paper in your agent:
hf papers read 2607.03738 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper