As always, I am appreciative of your comments. They always make me check my work.

Short answer: once, at startup. Longer answer: I went and re-read my own gates instead of answering from memory, and it's worse than I would have told you.

The one gate I had that was actually exact (a plain != check, no tolerance, since int8 x int8 into int32 has no rounding to hide in) was sitting on a kernel my transformer stopped calling when I switched to block-scaled quantization. It still ran at boot. It still passed. It still printed "HW verified vs units" in the UI. It was verifying a code path that does no work. That's worse than having no gate, because it looks like coverage.

Why my live kernels only had tolerance checks is the part I think you'll appreciate. Fusing the dequant epilogue into the kernel is what destroyed the exact oracle. The int32 accumulator gets multiplied by the block scales in f32 inside the kernel and never comes back to the host, so there was nothing exact left to compare against. I bought throughput and paid for it in observability, and I didn't notice the invoice until you asked.

The fix was to emit two variants of each shader from the same source string, where the only difference is the final write (one does the epilogue, one dumps the raw int32). Same indexing, same instruction, so gating the verify variant actually gates the real kernel. Then the surprising bit: once I made my JS mirror round the way WGSL rounds (f32 after the int cast, f32 after each multiply, instead of doing the whole chain in f64), the epilogue passed an exact check too, on real hardware. The tolerance was never covering hardware variance. It was covering a bug in my own mirror.

But the part you were really getting at is the one I had backwards. I'd been thinking of the weight hash as a partial backstop on kernel correctness. It isn't one at all. Weights only depend on the gradient bytes everyone receives, so if one device has a broken kernel it broadcasts a bad gradient, everybody averages the same bad bytes, and every replica stays bit identical and perfectly happy. There is no kernel disagreement the weight hash can detect. It's not a weak check on that axis, it just isn't a check on that axis.

So there's a canonical probe now. A fixed seeded GEMM run through each device's live kernel, hashed on the raw int32 accumulator (exact on every backend), sent with every gradient and re-run every 25 steps. Same input plus correct kernels means the same number whether you're on a phone GPU or a CPU fallback. My CPU mirror and my GPU DP4A path both come out to 2186402845, which is honestly the first evidence I've had that those two agree. There's also a random cell audit that re-checks live GEMMs against the units mid run at the real shapes, since toy shapes at boot were never going to catch thermal or shape dependent drift.

Then I did the thing your corpus implies and mutation tested my own gate. Five injected bugs (dropped batch stride, short K loop, missing column scale, dropped ReLU, wrong rounding) all rejected. The old allclose gate passes the rounding one that the exact gate catches, so your point reproduced itself in my own code. Live fault injection: 1.08e-7 drift, an order of magnitude below the old tolerance, caught mid run at 256x32x32.

Last time I thought the weight hash covered kernel correctness. It doesn't, it covers replica agreement. Now you're pointing out the probe covers backend agreement, and I called that correctness too. Same mistake, one floor up. I keep looking at two things matching and reading it as two things being right.

The referee question is the good one, and the answer is: me. Which is not a great answer.

In my defense, for about one second, the rounding fix wasn't quite "shader disagreed so I changed the mirror." The WGSL spec says f32 rounds after every op, and my mirror was doing the whole chain in f64 and rounding once at the end. So, the spec is what told me the mirror was wrong. The GPU just happened to agree. But nothing in the repo enforced that. It was me, with a red test in front of me, wanting it green. The difference between "spec says the mirror is wrong" and "shader says the mirror is wrong" lived entirely in my head, and I know which one I'd reach for on a bad day.

One small pushback. The mul8 table isn't a reimplementation, it falls out of the Python side as a trained artifact, and the DP4A kernel never touches it. So, table vs hardware is a real differential test with a referee neither side wrote. That part I think survives. What doesn't survive is the loop around it. The indexing, the strides, the accumulation, all of that came out of the same head as the shader on the same afternoon. Two implementations, one brain, same blind spots. You're right and I've got nothing.

So, I went looking for something that doesn't need a second implementation at all, and it turns out you can just use properties. A zero row of A has to give a zero row out. Permute the rows of A and the output rows permute the same way. A batched call has to match running each one on its own. None of that comes from anyone's version of the thing, it comes from what the thing is. And when one fails there's no referee argument to have. It's just wrong.

They bite, too. The batch one catches a dropped stride with nothing to compare against. The permutation one catches swapped rows. And a bug sitting in shared WGSL, the exact thing the probe can't see, can't hide behind agreement, because every replica breaks the property at the same moment.

Catch is they're partial. My zero-row property walks straight past the row-swap bug that the permutation one catches. You only cover what you thought to write down, which is your original point again, one floor further down. This building has a lot of floors.

Next up is your actual tiebreaker: an IEEE-754 oracle done in exact integer math, straight from the standard, so the spec is something that runs instead of something I remembered right once.

Doesn't fix the author problem though. I wrote the kernel, the mirror, and the properties. That's what an outside corpus does that nothing in my repo can. Asking again in case it got lost in the noise the first time.