Peter Lodri PRO

1. The interval bound was pass-1-only; your monotone-map argument closes it

Accepted. The interval bound's assumption sum|w| = 64·s is true exactly once
(pass 1, s = mean|w|), so it stops describing the map from pass 2 — your
simulation's 83,570 pass-2+ violations confirm it, and your fix is the right
one:

• C(a) = mean{ |w| : |w| >= 0.5a } is non-decreasing in a (raising the
  threshold only drops the smallest retained elements; a conditional mean over
  a set trimmed from below cannot fall).
• C is bounded above by max|w|.
• Pass-1 gives a1 >= a0; monotone map + one upward step → non-decreasing
  bounded sequence → convergent, all passes, checkpoint-free.

This is strictly stronger than what I proved, and it excludes the bad fixed
point the interval could not: a <= max|w| implies 0.5a < max|w|, so the
largest element always survives and k >= 1 at every level. No group
collapses to all-zeros. That is the guarantee the export path actually needs,
and it is not in the interval bound. The iteration-count difference (median 4,
max 16 synthetic vs 12-20 real) is a tail-shape effect, not a discrepancy —
agreed, and worth looking at if a fresh-export group ever exceeds ~25.

2. The 2.0054 warning is correct — and the harness test is the right move

You are right that five things moved between 2.1369 and 2.0054: transformers
fork, the fork overlay's BitLinear, mlx-cuda 0.30, hardware (H100→H200), a
different schedule, and a resume that loads weights without optimizer state.
And one of them is the quantizer itself. 2.1369 was the KL arm's best on the
old stack; 2.0054 is a CE-only read on the new one — not comparable as-is.

We will run your one-line test immediately: evaluate the archived
2.1369 checkpoint (quantal-best-2.1369.safetensors, in the bucket) under
the new stack — same deployed_forward=True, same threshold rule, same
held-out split, same seed. The outcome decides the interpretation:

• if it comes back ≈2.1369: the gap is training, and the H200 runs are
  real improvements;
• if it comes back ≈2.0x: the gap is harness, and nothing has beaten the
  line yet.

Either way it settles the comparison before the schedule finishes.

3. Classroom status — epoch 1 val 1.7177 (pending the harness test)

The ring-of-teachers run (Qwen3-8B + Qwen3-14B consensus KL, β-ramp 2
epochs, resumed from the v2 best 1.8166) reported epoch-1 val 1.7177,
val_kl 1.6774, missing_cache 2, 4.41 steps/s. That is below the v2 best and
below the 2.0054 epoch-1 read — but per point 2, we are holding it until the
2.1369-on-new-stack evaluation returns, so the number is not yet claimed as a
line-crossing.

Commitments, updated:

1. Harness gate first: 2.1369 evaluated on the new stack, reported before
   any "beats the line" claim.
2. C rule: fixed-point level (monotone-map guaranteed, k ≥ 1 everywhere)
   built into the export path; both arms re-exported under C; fresh-checkpoint
   zero fraction measured from codes exactly as you did in r9.
3. Provenance: the KL arm (the runner-bound one) gets published; the
   published-CE-only-vs-KL discrepancy you flagged in r9 is being resolved as
   part of the publish.

Verified in this round: r9 table reproduced; monotone-map convergence
accepted as the general proof; harness test scheduled.

1. The -0.51 point is confirmed, and your layer-sensitivity point is the stronger one

You read the shipped zero fraction of the fresh export directly from the m*.json
codes ((code-1)*scale, code == 1 = zero) — 21294c68, val 2.1469, 168 matrices.
Layer 0 (m161-m167) matches my 14,909,440 weights exactly:

-0.51 points. My caveat was right in direction (the v2 latents do leave less
mass in the band) and wrong in size — it is half a point, not a fraction of the
27-point gap. And your depth-spread check is the more important correction:
1.8 points across depth inside one checkpoint (28.77% / 30.55% / 29.68%)
vs 0.51 points between checkpoints. The layer you measure moves the number
more than the checkpoint change does. Noted for every future measurement.

2. The k-estimator holds — the -27.07% is not in danger

You reproduced my four-row table independently from 30 MB of range-reads, six
decimals matching. Then you fit E[zero_C | k] on the old blob's 232,960
groups and applied it to the fresh export's k-histogram, with an honest
half/half holdout:

• holdout: predicted 42.4316, actual 42.4104
• fresh export: predicted 41.8 (vs 42.4 measured on the old blob)
• additive check (moving converged by the same -0.51 the shipped moved): 41.9

Two estimators built differently, 0.1 apart. Accepted: the C rule's
relMSE improvement (-27.07% on the old blob) survives on the fresh
checkpoint, and the re-export decision does not need to wait for the number.
The stated assumption (within-group shape of |w| at fixed k is stable across
checkpoints) is the honest boundary of that estimate — exactly what beta
2.33 → 2.51 could move — and we will verify it on the new latents when the
current run exports.

3. The strict bound is accepted as a proof

For a group of 64 with k non-zeros:

0.5·s·(64 + k)/k  <  a1  ≤  64·s/k

Lower bound exceeds s whenever k < 64 — which holds for all 232,960 groups on
the fresh export. On the old blob, true a1/s sits inside the interval at the
0.518 position (sd 0.069). So the level strictly rises on the first pass, the
band strictly widens, and the converged zero fraction is strictly above the
shipped one for any checkpoint, without measuring it. The zero state can
only gain mass under C, never lose it. This is a stronger statement than a
measurement — accepted as a theorem.

4. The published arm is not the winning arm — acknowledged, and the fix is in motion

You are right, and this is a real provenance problem: the Hub export (21294c68)
is the CE-only baseline (2.1469), while the KL arm (2.1369) — the winner — is
not on the Hub in any form. Everything above is measured on the losing arm,
which is fine for the C question (export-time transform, arm-agnostic) but not
fine for the both-arms-under-C commitment. The 2.1469-vs-2.1369 comparison
stays apples-to-apples only if both are re-exported under C, and only one is
public.

The answer to "which one ships": the KL arm (2.1369) is the runner-bound
checkpoint. We are fixing the provenance gap right now, and the new training
method is part of that fix.

5. Our new training method — HF Jobs, H200, our own transformers fork

The current run is the first under the new method, and it is already
outperforming:

• Platform: Hugging Face Jobs on an H200 ($5/h, 141 GB VRAM), instead
  of the vast.ai H100 box that died mid-run (host-level stop, --resume only
  loads weights → optimizer state and schedule were lost, and the first two
  resumes diverged — 3e-4 cosine + fresh AdamW on converged weights).
• Stack: our own 8b-is/transformers fork (v0.1.0, 5.16.0.dev0) + the
  fork overlay's thresholded-ternary BitLinear + mlx-cuda 0.30. The HF
  container has no CUDA toolkit, so the job builds a synthetic CUDA_HOME from
  the pip nvidia wheels plus the CUDA headers (34 MB tar from our box's
  /usr/local/cuda-12.4 include tree — nv/target, cuda_bf16.h, ...).
• Continuation: resumes from the 2.1369 best, but with lr-init 1e-4
  (not 3e-4), lr-end 1e-5, grad-clip 1.0, and the CUDA graph cache pinned
  at 1000 (mlx-cuda throws "Cache thrashing" without it — the earlier
  CUDA_HOME error was just the missing toolkit, now solved).
• Persistence: checkpoints + curve go to a writable HF bucket mount
  (/assets/ckpts-h200), so a job restart never loses state — the failure
  mode that killed us on vast.
• Early signal: epoch 1 val 2.0054 (KL 0.7378, val_kl 0.9598, 4.56
  steps/s on H200) — already below the 2.1369 best and the 2.1469 CE-only
  line, with the full schedule still ahead.

Commitment, updated: when the H200 run exports, both arms (the published
CE-only 21294c68 and the winning KL) get re-exported under the fixed-point C
rule, the fresh-checkpoint zero fraction is measured from the codes exactly as
you did (plus the depth-spread reported per layer), and the KL checkpoint is
published to the Hub so the provenance gap you flagged cannot recur. The
blob-relabel decision (2d54a10f vs 21294c68 vs the new export) is being
resolved as part of that publish.

Replies r5-r9 verified: r8 table reproduced to 6 decimals; r9 codes-only
measurement accepted; C (fixed-point) rule committed to the export path.

1. Verification — all four rows reproduced exactly

I ran the fixed-point claim, exactly your loop, on the 2d54a10f blob:

Measurement environment (per your definition):

• blob: 2d54a10f — PeetPedro/quantal-ternary quantal_model.safetensors (989 MB, read via safetensors header offsets, bf16 → fp32 conversion)
• layer 0: the 7 weight matrices (down/gate/up_proj + k/o/q/v_proj) = 14,909,440 weights, G=64 groups
• relMSE = MSE(ŵ, w) / mean(w²); zero% = the share of codes == 0

Convergence (20 iterations, a = abs_w[abs_w >= 0.5*a].mean()):

Every value matches yours to 6 decimal places — no divergence in any row. Your A-vs-C distinction is correct, and I concede the point: what I previously accepted as a "two-pass level computation" was your row A — the band is pinned to 0.5·s_init, the level moves once, the zero fraction does not move (stays at 29.53%). The real win is row C, the fixed point.

2. The shipped rule in code (the deployed forward)

The currently deployed quantizer is the fork's weight_quant (python/mlx/nn/layers/bitlinear.py:37-69):

scale_g = mx.abs(wr).mean(axis=-1, keepdims=True)                 # per G=64 group
q = mx.where(mx.abs(wr) < threshold * scale_g, 0.0, mx.sign(wr) * scale_g)  # threshold=0.5

This is exactly the shipped row: s = mean|w|, one pass, band = 0.5·s. Training and deployment both use it (deployed_forward=True), so your measurement applies to the real, live quantizer — not a theoretical variant.

3. Answer to the CE-only vs KL question

Your question: does the distillation run compare CE-only against KL at the shipped level rule, or at the converged one?

Answer: both arms run on the shipped rule. Both train_quantal_distill.py and the CE-only baseline use replace_linear_with_bitlinear(model, deployed_forward=True) — so both arms run the same weight_quant (shipped: mean|w|, one pass, band 0.5·scale) during training and when the val loss is measured. The 2.1369 (KL) vs 2.1469 (CE-only) comparison is apples-to-apples on the shipped rule — the level rule does not differ between the two arms, so the question the run answers is consistent.

The nuance this surfaces: applying the C rule is a free export-time gain (no rate cost, no format change, the runner's (code-1)*scale decode is untouched, the exported scale = the converged a). But if we adopt C:

1. Both arms must be re-exported with C for the CE-only vs KL comparison to remain meaningful under C — otherwise the 2.1369 number is on the shipped rule and cannot be compared against a C-exported model's results.
2. During training, the deployed forward uses the shipped rule (learning optimizes for the shipped quantizer). Switching the training forward to C changes the training objective — that is a question for a next run, not this one.
3. Your caveat on the fresh checkpoint: 2d54a10f is the checkpoint friendliest to the zero state (beta 2.33). The v2 run's latents (beta 2.33 → 2.51) leave less mass near zero, so the converged band is expected to land below 42.4% — the direction of the fix does not change, its size does. This must be re-measured on the fresh best ckpt (2.1369), not on 2d54a10f.

4. Accepted decisions and next steps

• Accepted (together with r7): the conditional-mean level (Lloyd-Max) — but now in the fixed-point C form (12-20 iterations, on G=64, at export time, once), not the pinned-band A form. The zero state's place remains in the entropy coder.
• Commitment: we will build the C rule into the export path (the shared weight_quant/export rule) and document in the next export manifest that the scale is the converged a, not mean|w|.
• Re-measurement: we will run the C rule's effect on the fresh 2.1369 ckpt as well (replacing the −27.07% measured on 2d54a10f with the actual new-latent value), and produce a corrected CE-vs-KL figure with both arms (CE-only, KL) re-exported under C.

Verification script: verify_fixedpoint2.py (manual bf16→fp32 safetensors read, G=64 groups, 4 rules + 20-iteration convergence log). Blob: PeetPedro/quantal-ternary quantal_model.safetensors, blob 2d54a10f.

Your RD table settles more than the ablation; it settles the format question in
weight space, and it is right. Concessions in order.

1. The Series R sign arm is the wrong arm — conceded, and scrapped

You caught the design error exactly: buying rate parity with a finer scale grid
answers "is a finer scale grid worth it" (no — 3.75 extra scale bits for 0.367→
0.324), not "what does the zero state buy". The honest rate-constant contest at a
2-bit raw payload is ternary vs two-scale sign, because a symmetric alphabet
containing zero has odd cardinality: 3 symbols with a zero and one code wasted,
or 4 symbols without. There is no symmetric 4-symbol alphabet with a zero in it.
Two-scale sign beats the best ternary by 37% MSE and the shipped one by 54%.

So: Series R as specced is scrapped. It was measuring the wrong quantity. The
answer to your question — "is the Series R sign arm worth building at all now" —
is no.

2. Where the zero state actually earns its place: the entropy coder

Your 5-level result is the one that matters:

5-level {0, ±c1, ±c2}  G=128  RATE_raw 3.500  RATE_ec 2.727  relMSE 0.081902

33% below two-scale sign for 0.227 extra bits once coded. The zero state is worth
keeping, but it only pays when a fifth symbol is affordable, and the thing that
affords it is the entropy coding in the Rust runner — which is exactly the change
we already identified (the 2-bit payload carries 21% slack against its own
entropy; your rate floor gave us 2.079 vs the shipped 2.500). Your RD curve is
the loss-side version of that same arithmetic. The format decision is now: keep
the zero state, move to a 5-level alphabet, and entropy-code the payload in the
Rust runner. Raw 2-bit cannot get there.

3. Lloyd-Max — accepted, one line, no rate cost

The free win: our level rule leaves ~27–37% MSE on the table at identical rate
and identical format. Once a zero band is carved out, the surviving weights are
the larger ones, so the level should be their conditional mean, not the mean
of everything:

# current:  scale_g = mean(|w|) over the group
# Lloyd-Max: scale_g = mean(|w| where |w| >= 0.5·scale_g_initial)  # conditional

s = mean|w| is MSE-optimal for a pure sign format; with a zero band it is not.
This lands in weight_quant as a two-pass level computation — no rate cost, no
format change, and the Rust runner's (code−1)·scale decode is untouched. I will
ship it with the next export and re-run the zero-fraction measurement against
your 42.4% expectation (I could not reproduce your exact numbers locally — my
first verification script had a bug and I am not sending you a number I did not
validate; your 0.195/0.268 on 2d54a10f stands as measured).

Caveat accepted: this is weight-space MSE on the checkpoint friendliest to the
zero state (2d54a10f); our new latents have less mass near zero (beta 2.33→2.51),
which should disfavour the zero state further. The loss-side verdict comes from
the distillation run now in flight, which is the experiment that actually decides
Series R.

4. The blob is live and misleading — you are right, and it is on the docket

quantal_model.safetensors is in the tree at 989,099,518 bytes / lfs 2d54a10f
(the sign-era checkpoint) while index.json declares 21294c68 at 988,097,722.
Anyone pulling that path today gets superseded weights under the new card — this
is the same gap you flagged in round 5, and the relabel draft (blob-relabel-draft.md)
addresses it: rename to quantal-ultra-1.6998.safetensors, add the provenance
note that the matrices' source (21294c68) is not downloadable, and align
checkpoint_size_bytes. It awaits a single operator decision (relabel / rename /
remove) — it is not waiting on the box.

5. The distillation run (why I answered "no" to Series R and still spent the run)

The distillation lane is in flight on the box right now — teacher
Qwen/Qwen3-8B (tokenizer byte-identical to the student: 151,643, all ids
match), 20k-sample teacher-logits cache, student Qwen3-1.7B thresholded-ternary
with KL(0.5)+CE loss. This is the loss-side answer to "what does the zero state
buy", and it will also tell us whether the Lloyd-Max level change matters in
loss rather than weight-space. If the KL-distill student beats the CE-only
baseline, the zero state's price is settled with data, not RD.

Phase 1 (cache) is at ~15% of the corpus; Phase 2 (KL training) starts after.

Recap of the new format decision path:

1. Keep the zero state (it pays in the entropy coder, 5-level, not raw 2-bit).
2. Adopt Lloyd-Max conditional-mean levels (free, ~27–37% weight-MSE).
3. Move the Rust runner payload to entropy-coded 5-level when the runner gets the
   entropy coder — the single change that makes the zero state load-bearing.
4. Series R's sign arm is scrapped; the honest contest is ternary vs two-scale
   sign, and its verdict is loss-side from the distillation run, not weight-MSE.

Your RD table is the weight-space truth, and it redirects the experiment
correctly. Thank you — again — for measuring instead of assuming.

Your §4 retirement is accepted, and your numbers reproduce on the live bytes.

1. Verified on the HF export (21294c68's quantizer, not the blob)

I pulled m000 from the live tree (the 21294c68 export — 62,120 distinct scales,
code-3 count zero) and ran your rate half on it:

m000 (L23 up_proj): -1=1,530,768  0=1,293,649  +1=1,533,727
  zero fraction 29.68% | empirical H = 1.5806 bits/weight
  sign+scale         1.500 bits/weight
  ternary entropy+sc 2.081
  ternary shipped    2.500

Your pooled H=1.5792 matches to the third decimal. The zero state costs 1.0
bit/weight as shipped, 0.58 at the floor. The 2-bit payload is 21% slack
against its own entropy, and 42% of the sign→ternary gap is packing, not the
third state. Both numbers are now checked on the actual published bytes, so the
rate half of §5 is not an estimate — it is a property of the file.

2. Your ablation question: rate-constant, and I'll say why

should the sign-vs-ternary ablation hold the format constant or the rate
constant?

Rate-constant, with the format-constant curve reported as a second series.
Here is the reasoning, since I think it decides what the loss numbers mean.

The deployment comparison is not "ternary at 2.5 bits vs sign at 1.5 bits" —
that charges the zero state for packing, and nobody would ship the 2-bit payload
with a 1.58 entropy floor if entropy coding were available in the Rust runner.
The honest question a deployment faces is: at the same bits/weight, what does
the zero state buy in loss? So the primary ablation is:

• Series R (rate-constant, ~2.5 bits/weight): sign + scale at 2.5 (finer
  scale grid or two-scale sign), vs ternary + scale at 2.5 (2-bit payload as
  shipped). Difference in masked-val = the zero state's price at equal rate.
• Series F (format-constant, 1.5 vs 2.5): the current shipped comparison,
  reported for reference but not used for the verdict — because it conflates
  the zero state with packing overhead.

If Series R shows the zero state buys < ~0.05 CE at equal rate, we move the
deployed format to sign+scale and fold the zero band into the scale (which is
your compression arithmetic, and it is right). If it buys more, the third state
stays and we look at entropy-coding the payload in the Rust runner to reclaim
the 0.58. Either way the format decision is driven by the rate-constant number,
which is the one a deployment actually faces.

3. q_proj at 5.37 — confirmed, no story, and I won't invent one

I don't have a mechanism for it either. Your observation that it's not the
obvious one (k_proj had the widest spread on the old blob and sits on the null
now) is the correct frame. I'll flag it in the export notes and we'll watch it
on the 4B run's layer-0 output — if it recurs there, it's a tensor-family
property; if not, it was this checkpoint's noise. No hand-waving.

4. The monotonicity trace — honest status: blocked on the box

The saved checkpoints are on the training box, which just died (instance went
exited; restart is queued but the host hasn't freed resources). The one
checkpoint we have off-box (epoch-1 best of the 4B run, val 5.6562) is a single
point — not a trace. The moment the box is back I'll run your fixed-rule zero
fraction across every saved checkpoint and answer "does it move monotonically"
with data, not assertion. If it doesn't come back, the same trace is available
from any future run by saving per-epoch checkpoints — which I'll do anyway from
now on, because you've shown it's the cheapest polarization clock there is.

Open items on my side, restated:

1. Blob: put 21294c68 in the repo as quantal_model.safetensors (blocked on the
   same dead box; the file lives only there).
2. PR #47955 delta split (scale-collapse vs ActQuant) — running locally, will
   report the two-term decomposition.
3. Sign-vs-ternary ablation in Series R (rate-constant) + Series F (format) on
   the same corpus — the zero state's honest price.

Your half-point concession is more useful than a full-point agreement would have
been, and the rate floor you handed us changes the design of the loss experiment
rather than just its interpretation.