Party is over: regularizing ColBERT models to fix efficient ANN methods

Efficient ANN methods like MUVERA and SMVE promise to simplify ColBERT infrastructure, but they fell short with modern ColBERT model. We trace the root cause to embedding geometry, show that mean-centering is a strong but insufficient fix, and introduce STE-based regularization that directly optimizes the model for the target projection space. The surprise: the fix doesn't make embeddings more isotropic but concentrates them into fewer dimensions, the opposite of what intuition predicts, but what random projections need. The regularization transfers across methods, seeds, and hyperparameters without degrading full MaxSim retrieval.

🤗 Models: LateOn-regularized | LateOn

Table of Contents

• Late Interaction: Powerful, but Expensive
• The Solutions
• Are You Sure About That?
• Anisotropy, the Ideal Suspect
• The Easiest Way to Optimize Something Is… to Optimize It
• Talk Is Cheap, Show Me the Results
• What Did the Regularization Actually Do?
  • I. Anisotropy Was Not (Only) the Problem
  • II. Fewer Effective Dimensions, Not More
  • III. Learned Concentration vs. Naive Compression
• Takeaways and What's Next

Late interaction: powerful, but expensive

Late interaction models (also known as ColBERT or multi-vector) have exhibited very strong performances on various retrieval tasks such as out-of-domain, long-context, code and agentic retrieval. Despite these strong results, they come with infrastructure challenges because they require storing and searching all the embeddings corresponding to each tokens in the documents. Besides the storage cost, it also relies on index structure to generate a candidates set before gathering and decompressing the document token embeddings and applying the full MaxSim operation. This is typically handled with PLAID indexes, which we made effortlessly and efficiently available to the broad audience through FastPlaid and NextPlaid.
Although those libraries enabled the use of ColBERT models at larger scale, PLAID indexes can still be complicated to maintain at scale. Notably, updating the centroids remains a challenge due to the limitations of IVF-PQ that would require to rebuild the whole index once the distribution shifts to much, even after adding some mechanisms to follow that shift in FastPlaid ; which makes CRUD operations difficult. SPFresh offers some trade-off to make updates easier, but might face other scaling issues.
For these reasons, new methods have been proposed to generate candidates without relying on centroids.

The solutions

MUVERA uses the multi-vector representations to create Fixed Dimensional Encodings (FDEs), akin to large, dense, single vectors to leverage existing single vector infrastructure such as HNSW.
SMVE use random rotation projections to create sparse encoding of the multi-vector representations to leverage existing sparse retrieval infrastructure.

Those methods are very attractive because they would simplify the multi-vector retrieval infrastructure extensively by offering easier to maintain and CRUD-friendly indexes to generate candidate sets to be reranked with full MaxSim. However, the early experiments exhibiting promising results were run with the old ColBERT-v2 model, and surprisingly did not seem to generalize to newer models…

Are you sure about that?

On the 27th August 2025, an issue was opened on PyLate, describing that ModernColBERT was not performing correctly when used with MUVERA. Quickly, we realized that this issue extended to other models such as ColBERT-small, which excluded our training loop from the suspects because it was not trained with PyLate. We then further hypothesized that it could be due to the scores normalizations in distillation setup, until we ruled it out by performing KD without normalization and then further confirmed that some contrastive-only models such as ColBERT-Zero-supervised also suffered from the issue. Since most of the new models did as well, it became clear that it was not really tied to a specific training setup but was a more global issue. A simple explanation would have been that those methods simply were not working properly; but why would it work for some models (including new ones) and not for others?

Anisotropy, the ideal suspect: if everything is similar, nothing is dissimilar

Then, someone came up, asking:
> Question/curiosity - why are GTE-ModernColBERT-v1's token vectors so much more anisotropic than colbertv2.0's?
> Mean pairwise cosine similarity for GTE-ModernColBERT-v1 on FreshStack datasets is ~0.9. For Colbertv2.0 it's ~0.2.
At that time, the answer seemed to be that it was probably related to the training setup, that also caused MUVERA to fail. However, it turns out it could be a key clue about why MUVERA is failing.

When we think about embeddings for retrieval, we love to imagine them scattered evenly across the hypersphere (i.e, scattered around the space), so that the proximity in the space is meaningful, we can divide the space into cells to perform search only a subset of those, to be able to quantize the embeddings without introducing errors by collapsing two nearby embeddings, …

However, in the case of most recent ColBERT models, they collapse into a narrow cone, all leaning in roughly the same directions (as indicated by the 0.9 mean cosine similarity). What it means is that most of the token embedding energy is just a single shared axis (essentially the corpus mean) and only a thin residual actually encodes the useful, discriminative information that is actually useful for retrieval.

This could explain why MUVERA and SMVE come short with those models. Instead of relying on centroids (that would “eat up” the common direction), MUVERA partitions the space with random hyperplanes (à la SimHash) and pools the tokens that fall into each bucket into a Fixed Dimensional Encoding, while SMVE applies random rotations, keeps the largest projections per direction, and sums them into a sparse sketch. But random partitions only separate points that are actually separable in the space. When all of the embeddings are within the same thin cone, a random hyperplane almost always split the space with the whole cone on one side and a random projection returns the same value for every embedding. This means applying these approaches to anisotropic ColBERT just yields degenerate outcomes: every token maps to the same code and thus every document maps to nearly the same FDE/sparse vector, making the candidate generation step essentially random.

The exact reasons why the new ColBERT models are so anisotropic remains unknown and are surely an interesting avenue of research. Among the running hypothesis, we know that ModernBERT base embeddings are somewhat already anisotropic which does not help (as most of the modern ColBERT models are based on this backbone), but this cannot be the only explanation as ColBERT-small is not ModernBERT-based. Another possible factor is that, in ColBERT training, we only directly backpropagate into the tokens that have been selected by the MaxSim operator. While this helps stabilize the training, this also means that a lot of the tokens' training signal is very indirect and tokens are just “pulled” by the other tokens' updates. Finally, it seems that in all generality, embedding models tend to become anisotropic.

One simple post-hoc solution is to center all of the embeddings by subtracting the mean direction (either mean without the document or the collection). This obviously reduces anisotropy by a very large factor and, by centering the cloud, the random projections are much more likely to split it across the informative axis. And this is not only theoretical: simply centering the embeddings before computing MUVERA/SMVE representations allows LateOn (our latest state-of-the-art ColBERT model) results to go from plain unusable to somewhat working but still far from the target. This is because this is only partially solving the problem: we remove the majority of the noise coming from the shared common direction, but the cloud is still not correctly shaped to efficiently be projected into those representations. Although the latent anisotropism is wiped from the embeddings, they are still not efficiently separable through the random operations.

This does not mean that this is a fatality and post-hoc methods only go so far, we are limited by the space learned by the model.
This is why regularization objective that attempt to spread the vectors across the hypersphere have been proposed for a long time and are still very much used, notably through the GOR objective introduced in 2017 has been used in very recently in GemmaEmbeddings and Jina Embeddings to enhance quantization robustness.

The GOR objective is as simple as it gets: we don’t want two embeddings to be similar, so we will take the pairwise cosine similarity matrix between all the embeddings (Gram matrix) and try to push all the non diagonal terms to 0. This effectively tries to minimize any similarity and create an isotropic space.

Thus, it seemed very obvious that the solution was just to regularize the space when training ColBERT models to prevent them from using the space suboptimally, which is why we started adding GOR in the training. However, contrary to dense models where it just makes sense to push all the (negative) in-batch vectors towards different parts of the space, the multi-vector setup introduced some questions. A negative document can contain tokens that should actually be similar to the token of the positive document. It makes sense that the word cat is represented similarly, although the two documents are not covering the same exact topic. Take the example of “the cat drinks milk” and “the cat drinks water”. We don’t want the embeddings of “the” “cat” and “drinks” to be radically different in the two examples, we want the distinction to be made by the “water”/”milk” token. This also makes sense after the projections: we want most of the codes of the both representations to be similar because those are similar sequences, and so they should be close in these projected spaces as well. This is achievable if most of the tokens are similar (and thus receive the same codes), and only differ by the codes corresponding to the difference in the last embeddings.
Also, considering that our problem is document embeddings anisotropy, should we also regularize the queries or let them be trained by the document regularization?
While trying various different setup using all the pairwise similarity matrices possible and trying to balance the different weights to fix the issue without degrading the main performance of the model (without that much of a success), a question arose… should we even bother?

The easiest way to optimize something is… to optimize it

Everything we’ve discussed so far is theories and hypotheses based on some observations. Until now, we have no certitudes besides that the embedding space is not suited and that we should regularize. What to regularize and how are still very much open questions.
So, instead of guessing, could we simply optimize the model to work great in our target representation space?
The main issue is that we can’t simply apply the contrastive loss onto the projected codes and hope the gradients sort it out… because the creation of the representations blocks them. More precisely, the discretization step of both pipelines breaks them.

In MUVERA, this step is the SimHash partition: each token is routed using the sign of its projection onto a few random hyperplanes, and those signs are stitched into a one-hot assignment to one of the buckets. But sign is flat on each side (−1 / +1) and only moves at the boundary, so moving within its cell changes nothing; the derivative is zero, and it's that which-bucket assignment that is opaque (the pooling that follows is a harmless sum).
For SMVE, after computing the random rotations and the associated projected values, we keep only the largest projections per direction and zero the rest to make it sparse. Which values survive depends on the ordering and thus, small changes are not reflected until the order changes, making the gradient zero again. In both cases, the gradient-killers are hard, piecewise constant selections.

If you have trained quantization-aware models, it might ring some bells as this is the exact same issue. Quantization transforms a continuous value onto a discrete grid of levels, rounding it to the nearest level and creating a staircase: flat between levels with zero gradient almost everywhere except at the boundaries. One very popular approach to perform Quantization Aware Training (QAT) is by using the Straight Through Estimator (STE). The idea is quite simple: we perform the gradient-breaking operation in the forward pass and compute the loss value as usual. The trick is during the backward pass: we pretend the hard selection operation was actually an identity operation gradient-wise and let the gradient flow as if nothing happened.

In our case, we build the true MUVERA/SMVE code and score it using contrastive loss, so we are actually directly optimizing the representation we’ll then leverage at test-time. On the backward pass we treat the bucketing/top-k operation as identity on whatever it selected: each token receives the gradient of the bucket it landed in, and SMVE passes gradient through the surviving coordinates and zero through the dropped ones. The linear machinery around the selection (projections, rotation, pooling) keeps its exact gradient.
The loss gradient therefore flows straight back into the embeddings, and the model is trained to make its codes discriminative after the projection, optimizing the target metric directly instead of a hand-crafted proxy. Centering and GOR were us guessing at what a well-shaped space should look like and hoping the projections would cooperate; STE lets the projection itself tell the model what to fix.

It's worth noting this is obviously not free: the straight-through gradient is biased as the gradient we pretend exists is not the real (zero) derivative, and the term we drop is exactly the one that would push a token across a hyperplane or flip the ranking. This approximation can be noisy, but since we use it only as an additional regularization signal alongside the standard MaxSim contrastive objective, we still learn a genuinely good late-interaction space while pushing it toward something that projects cleanly.

Talk is cheap, show me the results

As we found out centering to be a very strong and cheap operation, all of our trainings and evaluations are run using mean-centered representation. The training setup is pretty simple: we take the unregularized unsupervised model and perform the same supervised training step, but adding the additional gradient coming from STE as a regularization. The loss is (1-alpha) * MaxSim contrastive loss + alpha * projected representations contrastive loss. Note that this supervised stage is very cheap to run (we did not even run the full 3 epoch) and thus show the results that are obtainable with a very lightweight regularization.

The first very clear result is that these very simple regularizations achieve their goal and significantly boost the performance of SMVE/MUVERA over the unregularized (LateOn) model without degrading the PLAID results (meaning that having the space correctly shaped does not conflict with the MaxSim requirements). The base results (without any reranking) are much stronger by themselves, but it also shows that the gap with the full PLAID retrieval can be smaller when coupled to top-k reranking. Please note that we did not optimize the hyper-parameters of MUVERA/SMVE to boost performance. It is possible that using enough dimensions, we could totally close the gap (even with fewer reranked elements), but the goal of this study was simply to show that the models could be regularized, not study the trade-off between cost/performance of all of these methods.

The second (and probably most important) thing to note is that both regularizations help both methods: using MUVERA representations during training helps SMVE during evaluation and vice-versa. This is a critical piece of information that shows that the training did not optimize only one specific subspace, but improved compression globally. Indeed, this was a very important verification because it is worth mentioning that, during training, the random elements are kept fixed (sampled once at the start of the training). We tried randomizing it at every step, but it seems that it made the training unstable because the signal from ever-changing spaces was confusing the model. Training for various bases by alternating every few steps could be a direction, but we pleasantly found out that the model was not overfitting to these fixed elements. Although the two methods are vastly different and we used different seeds for both anyways, we validated the results using various test seed for MUVERA, and the results show that there is very little variance and, most importantly, the training seed is not the best of it all, totally ruling out space-specific learning.

We can see that the variance is not too big and is even smaller when we consider reranking. Interestingly, we can note that the seed used for the training (1221) is not even the strongest one of the 4 seeds evaluated, emphasizing that the variance comes more from the methods themselves rather than a specific overfitting.

Besides specific projection directions, the model could also overfit to the specific MUVERA hyperparameters used during training (R=20, k=5, d=16, FDE dim=10,240), making the regularization ineffective at other operating points. To test this, we swept each MUVERA parameter independently at evaluation time while keeping the others fixed. MUVERA builds a fixed-dimensional encoding (FDE) by first partitioning token embeddings with SimHash, then pooling within each partition, projecting each pooled block to a lower-dimensional space, and finally concatenating the results across several independent repetitions. In this construction, (R) is the number of independent repetitions, (k) is the number of SimHash bits used to define the partitioning, and (d) is the per-bucket projection dimension after the inner random projection. Together, these parameters determine the FDE size: FDE dim = 2^k × d × R

Across all configurations, from aggressive compression (FDE dim=1,024) to high-capacity (FDE dim=20,480), the regularized model consistently outperforms the baseline. Notably, the improvement is largest away from the training configuration: at FDE dim ~1K the rk=0 gap reaches +7–13 NDCG@10 points, compared to +8 at the training dim of 10K. This confirms that the regularization teaches a general geometric property rather than optimizing for a specific set of projection parameters. Exploring whether training at a different capacity, e.g., smaller FDE to force stronger compression, or larger to relax the constraint, could further improve results is an interesting direction, but orthogonal to our main finding and left for future work.

What did the regularization actually do?

Beyond giving us a model that works with MUVERA and SMVE, the STE training left behind a second gift: two checkpoints (before and after) that we can probe to understand what changed in the geometry of the space. We ran a battery of measurements on all documents and queries of SciFact, computing every metric on mean-centered representations. The findings are informative, and run directly counter to the narrative we built in the first half of this post.

I. Anisotropy was not the (only) problem (or at least, not the whole story)

Everything above pointed at anisotropy as the culprit. So we expected the regularized model to produce a more isotropic embedding space. It doesn't.

The regularized model is actually slightly more anisotropic than the original, in both the raw and centered spaces. The STE regularization, which dramatically improved MUVERA/SMVE quality, did not make the space more isotropic. If anything, it made it slightly more anisotropic.

This makes sense in retrospect as we trained on centered representations, so the model never received any signal to reduce the dominant mean direction. But it means that anisotropy per se is not the damaging property, or at least not the only one (centering does a lot of the heavy lifting here). It was a useful clue that led us here, but the real mechanism is elsewhere.

(Note: The Iso-ModernColBERT model effectively reduced the anisotropism of ModernColBERT, but in our experiments, it seems to brings improvements when indexing non-centered embeddings but is not much better when using centered embeddings, confirming that anisotropism is an issue but that is not the only issue and that centering is a fair post-hoc fix for this part of the problem. However, it's possible that centering actually hurts the results (masked by the large gains brought for MUVERA/SMVE) and that letting the regularization "learn" the centering might end up in a more optimal solution.)

II. Fewer effective dimensions, not more

While searching for what actually moved during the training, we measured the Stable Rank (SR), a smooth estimate of how many directions a set of vectors effectively spans. We computed it at two levels: per-document and corpus-wide.

Both stable ranks decreased. Per-document, tokens that spanned ~5 effective directions now span ~4. At the corpus level, the cloud contracts from ~15 to ~11 effective dimensions. The top eigenvalue captures a larger share of the variance, meaning each document's token information concentrates into fewer principal axes.

This is the exact opposite of what a GOR-type regularization would do. GOR pushes tokens apart to fill the space uniformly and increase stable rank. STE did the reverse: it collapsed each document's tokens into fewer dimensions. And yet this is what made SMVE and MUVERA work dramatically better.

(Note: the SR results are also interesting by themselves. It means we are encoding and storing various vectors while the actual dimensions are much lower.)

Why fewer dimensions help random projections. The SMVE score between a query and a document is a sum over a fixed number of random projection directions. Each term captures one random "glimpse" of the query–document interaction. When the token cloud spans many effective dimensions, those glimpses are spread thin: each captures a small, noisy slice, and the sum has high variance as an estimator of the true MaxSim score. When the cloud spans fewer dimensions, the same number of glimpses oversample the space more densely. Each captures a cleaner slice, and the sum converges to a tighter estimate.

The width sweep confirms this directly. We ran SMVE at different numbers of random projections (from 128 to 1024):

The regularized model wins at every width, but the advantage narrows as width increases. With enough random projections, even a higher-dimensional space gets adequately sampled, so the regularization matters less. This is exactly the pattern you'd expect if the mechanism better covers a lower-dimensional subspace.

III. Learned concentration vs. naive compression

If fewer dimensions help, a natural question arises: why not just project the embeddings onto their top PCA directions post-hoc to reduce dimensionality? We tested this by taking LateOn's tokens, projecting them onto the top-k corpus principal components, and measuring SMVE quality:

The results are clear: naive low-rank projection is catastrophic. Keeping only 4 PCA directions destroys 93% of SMVE quality; even 16 directions lose 85%. Simply squashing the space into fewer dimensions wipes out the retrieval signal along with the noise.

The regularized model achieves a comparable reduction in effective dimensionality, yet its SMVE quality improves by 85%. The difference: the model learned which dimensions to concentrate into. The STE gradient tells each token which projection bucket it landed in, so the model can rearrange its embedding space to place discriminative information along the axes that survive the projection rather than along arbitrary high-variance directions that happen to capture the most energy in an unregularized space.

Query–document alignment measurements confirm this. We measured how much query token energy falls within the top principal components of the document token cloud:

Queries concentrate more of their energy into the same principal components that documents use. When both live in a more shared subspace, random projections through that subspace capture retrieval-relevant signals on both sides simultaneously, making each SMVE/MUVERA term more informative.

This is why directly optimizing the target with STE is so much more effective than geometric proxies: the projection loss captures the full chain from geometry to score. Centering and GOR are educated guesses about what a well-shaped space should look like. STE lets the projection itself tell the model what to fix.

Takeaways and what's next

Four things to remember from this work:

1. New ColBERT models are highly anisotropic, which breaks ANN candidate-generation methods like MUVERA and SMVE that rely on random projections to separate embeddings.
2. Mean-centering is a strong baseline that removes the dominant shared direction, but it's not enough on its own as the residual geometry still doesn't project cleanly.
3. STE-based regularization fixes the problem, not by spreading embeddings uniformly across the sphere as expected, but by teaching the model to concentrate discriminative information into a low-dimensional subspace that random projections can efficiently capture, the opposite of what isotropy-based intuition predicts.
4. The regularization is generic: it transfers across methods (MUVERA-trained helps SMVE and vice-versa), across random seeds, and across hyperparameter configurations far from the training setup.

These findings also connect to a broader question about the structure of ColBERT embedding spaces. As we showed in our work on hierarchical pooling, much of the per-token information in a document is redundant. The stable rank results here reinforce that picture and suggest that the effective dimensionality of these spaces is much lower than the nominal embedding dimension. We believe this work is directly related to the recent line of work that learns SAE over ColBERT models to create sparse representations. Because MUVERA/SMVE are not learned, we have to make sure the representations are directly usable. We believe understanding the effect of this kind of regularization and uncovering the underlying structures in the embedding spaces that make them compressible (SMVE/MUVERA, pooling, quantization) and allow us to expose them through other types of indexes than just MaxSim could be very useful in building the next generation of models and indexes.

Get started now:

• 🤗 Models: LateOn-regularized | LateOn
• 🛠️ Tools: PyLate | FastPLAID

Citation

LateOn Regularization

@misc{chaffin2026lateonregularization,
  title={Party is over: regularizing ColBERT models to fix efficient ANN methods},
  author={Chaffin, Antoine},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/lightonai/lateon-regularization}},
}

DenseOn and LateOn

@misc{sourty2026denseonlateon,
  title={DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models},
  author={Sourty, Raphael and Chaffin, Antoine and Weller, Orion and Demoura, Paulo and Chatelain, Amelie},
  year={2026},
  howpublished={\url{https://huggingface.co/blog/lightonai/denseon-lateon}},
}