Note about Vision & Images
Tried your approaching when working with vision. So making this post as a sort of "word of warning" to anyone else walking this road.
Qwen3-vl-32b & Qwen3-vl-8b tend to focus different activation energy on entirely different tokens. So when one starts testing for rough correlation between 32b & 8b you'll find the distribution is strongly bi-modal. This is because AFAIT around L29 32b & 8b start "diverging" in what "KV/Tokens" they do/do-not "pay attention to" to paraphrase.
So a simple affine map from say 8b -> 32b straight up doesn't exist (with vision tokens). It has to have (basically) an attention mechanism to redistribute activation energy.
One can "very nearly" for an affine map at 8BL16 -> 32BL29, but if you include text tokens, doesn't work "that cleanly".
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
No. It means a smaller encoder is pretty fundamentally broken for handling image & video input without a secondary attention mechanism to "translate" the different learned attention patterns which Qwen3-vl-8b & Qwen3-vl-32b have encoded into them.
Text (and text alone) works fine, I have no clue why, the shared geometry is great. But multiple/mixed media plainly needs a non-affine activation aware transform.
Thanks for this — it changed what we trained on.
We are finalizing an MLP rather than a plain affine map. It comes out slightly better than the affine transform, but not dramatically so. The bulk of what a projection can recover seems to be within reach of a linear map already; the MLP buys refinement, not a different regime.
The concrete effect of your post was that we added reference images to the training corpus, which had been text-only until then. That turned out to matter a lot more than the choice of architecture. On 100 held-out images, the cosine on the conditioning improved from 0.71 to 0.85, and every single one of the 100 improved — no exceptions. Text quality did not regress.
My reading is that the map needs to see a diversity of token types to cover the space it is asked to fill. Vision tokens land in a region of the hidden space that a text-only corpus simply never visits, so the map was extrapolating there. It is not that images taught it something about images specifically — it is that they filled in a part of the domain that was empty.
Which is also why I am cautious about your bimodality point. I could not reproduce it on my side: measuring 32B against 4B hidden states on the same tokens, imposing the target's tokenization on the student so the sequences cannot drift, I get a unimodal distribution. That does not settle your mechanism argument, but it does mean the practical conclusion — that no affine map exists — does not hold: fitting one on data that includes vision tokens moves it substantially.
Where I think you are right is on the residual. What is left after the projection is exactly the kind of failure a position-wise map should have: small discrete attributes get lost — a hat, a colour — while motion, action and overall composition survive. A map that treats each token independently has no way to bind an attribute to the thing it belongs to.
Does it means the smaller encoder is better to use in scene like image to video, other than text to video, because the image provide more context？
No. It means a smaller encoder is pretty fundamentally broken for handling image & video input without a secondary attention mechanism to "translate" the different learned attention patterns which Qwen3-vl-8b & Qwen3-vl-32b have encoded into them.
Text (and text alone) works fine, I have no clue why, the shared geometry is great. But multiple/mixed media plainly needs a non-affine activation aware transform.
Oh, I thought that smaller encoder may not be able to capture abstract semantics, such names of people, etc. Using image to video, maybe this won't be a big problem, as we usually do not need to provide such prompts. I saw some comparison using image to video, the result is quite similar with the original encoder.
As the tests did not has complex scene, maybe more training on the small encoder is still needed.