Justin

They were measured in the same benchmark invocation per model. I went back to the archived runs rather than answering from memory.

In the production-scale benchmark, the KV caches and attention importances are precomputed once for the sequences. The FP16 reference is then calculated from those same cached PKVs before the config loop, and each config reuses the same pkv with base = fp16_bytes(pkv).

For Qwen2.5-32B specifically, the log has one shared 128.0 MB reference KV size, then:

K4V4 4.1x 30.9 MB
K4V2 6.2x 20.7 MB

Qwen2-72B is the same structure with one 160.0 MB reference:

K4V4 4.1x 38.8 MB
K4V2 ~6.1x 26.0 MB

So unless I’m missing something subtler in how fp16_bytes() behaves, I don’t think the two configs had different denominators. They are being compared against the same PKVs in the same run.

That seems to push the problem toward the encoded representation or enc.memory_bytes() rather than the baseline. I’ll inspect that next, but wanted to answer your invocation question first with the actual artifacts.

This narrows it down a lot. I hadn’t thought to subtract the configs first and treat the common terms as canceled, but the 0.42–0.78 bits/element excess makes the search space much cleaner.

The scale-cost observation is especially interesting. A ~0.5 bit/element config-dependent term landing that close to the historical K4V2 result is enough that I want to inspect the counter before spending time on the packer.

I don’t want to answer the group-size question from memory. I’m going to verify two things directly in the implementation: whether V2 and V4 use the same group size, and whether the scale/zero tensors are included in the counted representation identically in both paths.

I also like the incompressible-code test. If the serializer still reports ~6.1x on uniformly distributed 2-bit symbols, that pretty cleanly turns this into an accounting problem rather than a data-dependent compression explanation.

I’ll trace those two config paths and report back with the actual group sizes and byte components rather than speculate. Thank you for taking the time Dipankar, this is extremely helpful.

This is really helpful. And agreed on the negative class. I was focused on correcting the duplicate weighting, but one distinct negative scenario means the precision side is much weaker than the current metrics make it look. I’m going to fix that before the taxonomy gap: dedupe by scenario identity, add genuinely distinct negative fixtures, then regenerate the metrics.

On TSC, the fact that the two overshoots require different factors is something I hadn’t isolated. The ~4x difference in the implied missing bits makes the V path the obvious place to inspect.

I don’t want to answer the serialization question from memory and accidentally rationalize the old number. I’m going back through the implementation and tracing V from quantization through packing/serialization to the exact byte counter. I want to establish whether it’s truly stored at fixed 2-bit width, whether anything in that path can legitimately land below nominal width, and whether the reported byte count is measuring that exact serialized representation.

If the bytes can’t explain the gap, the 6.1–6.2x result stays retired.

Really appreciate you digging into this.

No, the duplication wasn’t deliberate. It looks like the generator expanded a smaller scenario template set and I failed to normalize scenario identity before aggregating the smoke metrics. Great catch. I’m going to fix the weighting around unique scenarios, add the missing taxonomy coverage, and regenerate the metrics. I’d absolutely take the per-scenario table if you have it.

And thank you for offering to read the paper. It’s actually on a separate research thread around Temporal State Compression rather than the Atlas. I put together the current external-review version here:

https://solsticestudio.ai/tsc-review

One reason I wanted another researcher to look at it is that the project has gone through a few rounds of self-audit where earlier claims were narrowed or retired when the accounting didn’t hold up. Given what you just found in the Atlas, I suspect you’ll have exactly the kind of criticism I’m looking for.

Hey Dipankar, noticed you liked my Agent Failure Atlas benchmark recently. Really appreciated that. I’m working on a related independent research submission and wanted to ask you something about arXiv. Is there a good way to reach you privately?