Join the community of Machine Learners and AI enthusiasts.
Sign Up
The suite answers how fast for 53 models and how good for 25, and the gap sits exactly where the interesting models are.
Counting rows in your own CSVs: 144 quantized entries across the perf tables, 2 across the accuracy tables (both Cosmos-Reason2). So every W4A16 LLM variant ships a throughput number and no quality number, and those are the ones carrying the win. Qwen3-1.7B goes 49.31 to 189.02 tps on agx_thor with FlashHead-W4A16. Llama-3.2-1B goes 77.30 to 284.76. That is 3.8x and 3.7x.
The headline question feels it hardest. The two smallest LLMs you measured, gemma-3-270m-it and Qwen3-0.6B, have no accuracy row at all. So how small can it go and still run is answered, and still be useful is the half not yet in the data.
Worth saying what the tables do prove, because it reads as under-sold: FlashHead is accuracy-neutral. Llama-3.2-1B-FlashHead matches the base model to two decimals on all seven tasks, and the 3B and Qwen3-1.7B are within 0.01 everywhere. A 1.26x with no quality argument to have is a rare thing to be able to show.
On use cases, mine is agent loops running on-device, where tokens/sec decides how many tool-call retries fit inside a latency budget. At that point a 3.8x is not a faster agent, it is a different agent.
Is the W4A16 accuracy sweep just queued, or is getting lm-eval through a TensorRT engine on device the actual blocker?
Yes, accuracy numbers are a bit thin! For FlashHead you are correct. We have written a paper on FlashHead (see https://arxiv.org/abs/2603.14591) where we show that it is consistently accuracy-preserving: typically within ~99.9% top-1 containment.
For the GenAI models on Jetson: We are actually using the TensorRT vLLM Docker image, so lm-eval has a path and is not blocked by the TRT engine. For example, for Qwen3.6 we have internal accuracy evaluations but not yet pushed.
For the more classical DL models, the benchmark suite mostly compares to quantized variants where we do need to push for more on-device accuracy evaluations.
The notion of picking the largest model that fits the latency (and memory) budget is quite common and that's one focus of our GenAI models . On the classical DL side it is a bit different, edge deployment use-cases are often around making an existing, well performing model faster.
The containment number covers the part that already had accuracy rows.
FlashHead is the one variant in your tables with a quality column, and it is clean: Llama-3.2-1B-FlashHead matches base to two decimals on all seven tasks. The 144 rows with throughput and no quality are the W4A16 ones, and those are the rows carrying the 3.7x and 3.8x.
Worth saying why containment is the right metric for a head and still not a substitute downstream. Top-1 containment is a per-step property and generation is autoregressive, so misses compound: 99.9% per token leaves about 82% of 200-token completions with no miss at all, under independence. Most misses will not change the sampled token and fewer will change the answer. But the failure is trajectory-shaped, and a per-step average is the one summary that hides it. That is an argument for your seven-task table, not against it.
The TRT vLLM image having an lm-eval path is the useful half of your answer. It moves the W4A16 gap from a tooling problem to a queue problem, which is a much better place to be.
If one accuracy row goes up next, which moves your own decision more: Qwen3-1.7B at W4A16, or gemma-3-270m-it where there is no quality number at all?
Good points! The order is a bit project-dependent though, and where the interest is strongest. From my point of view, Qwen3-1.7B-W4A16 is a bit more interesting as I see a bit more use around it and it can move an actual deploy decision.
But it is not necessary about which accuracy row lands next, but which model x hardware combination we optimize next. Being able to replace an existing deployed model fast with a newer, better one is quite a strong argument, or moving to a cheaper hardware.
In terms of what models may come next, I could see VLAs becoming more interesting going forward since low-latency real-time deployment scenarios are key there.
For example, we recently did a project focusing on VLA (Ο0.5) optimization on AMD. Some of this was in the AMD keynote last week. We did the GPU VLA inference-latency optimization behind their sub-100ms robot-reasoning number on the Kria AI SOM.
And FYI, missing accuracy benchmarks for the non-LLM/VLM models have been added now. π
Nine of twenty-one now, and the twelve still open are all the same shape.
Counting quantized variants across the CSVs: 21 carry a speed row, 9 now carry a quality row. When I ran that census for the first comment it was 2, both Cosmos-Reason2. Commit 93c3d46 added five families and b1cd606 added SAM-3D-Body. All nine classical families have a table now, and every one of the 12 still missing is a W4A16 LLM or VLM, which is exactly the queue you described.
The new rows are worth more than the gap they close, because quality-per-speedup is computable for the first time, and it does not rank the way the speed table does.
Chronos-2 is the outlier. 4.48 to 3.48 ms at ctx 2048 is 1.29x, and WQL over seasonal-naive goes 0.549 to 0.618. Every other classical variant costs at most 4.2% relative: MobileViT 4.2, SAM3 3.2, Parakeet 2.3 on WER, RF-DETR 2.2, paraphrase-MiniLM 1.5, DINOv3 0.5, all-MiniLM 0.3. Chronos is 12.6. And because that metric is normalized to seasonal-naive, the thing being bought is the margin: 0.451 down to 0.382. That is 15.3% of the skill over the naive baseline, for 1.29x.
Second thing the new rows expose, and it is not about accuracy. Parakeet on AGX Orin is 35.6 to 34.4 ms. 1.035x. Memory goes 1274 to 806 MB, 1.58x. That is not a weak speedup, it is a memory result filed in a latency table.
Which lands back on your own framing. You said the decision is the largest model that fits the latency and memory budget. mem is populated on 12 of 430 perf rows, 2.8%, and on 6 of 168 quantized rows. It happens to be filled on Parakeet, the one family where latency alone would say do not bother.
So the accuracy hole is mostly closed and memory is now the wider one.
Is mem there peak device memory or resident weights, and is it cheap enough to backfill across the table?
