resoajoe/quality-gate-nano

quality-gate — is this generated video frame usable?

47,122 parameters · 188 KB ONNX · 0.131 ms/frame on a Jetson AGX Orin CPU (7,641 fps)

SCOPE — read this before using it

What it is for: flagging catastrophically degraded frames in chunked video generation — frames where the scene has smeared, posterised, or lost chromatic coherence. It exists to decide "redraw this chunk or keep it" inside a generation pipeline.

What it is NOT for:

• Not an aesthetic or quality judge. It detects collapse, not mediocrity. A frame can be sharp, coherent and completely wrong (incorrect subject, impossible geometry) and pass.
• Not a general image-quality metric. It was trained only on interior-room scenes from two video generators. It has no calibration against human preference, IQA benchmarks, or any standard metric.
• Not a deepfake / AI-generated-image detector. Every training frame is generated. It cannot distinguish real from synthetic.
• Not validated on any domain other than the one described below.

Measured performance

Trained on 1,840 chunks from 34 generation arms across two model families (LTX-Video-2B and Wan 2.2 TI2V-5B), 16,320 frames, 36.1% collapsed.

seed	accuracy	majority baseline	lift	recall (collapsed)	precision
0	0.902	0.749	+0.153	0.672	0.913
1	0.941	0.586	+0.356	0.905	0.951
2	0.930	0.725	+0.205	0.940	0.962
mean	0.924 ± 0.017	—	+0.238 ± 0.086	0.84 ± 0.12	0.94 ± 0.02

Read the lift, not the accuracy. The class balance varies by split, so raw accuracy tracks the prior. Lift over the majority baseline is the honest figure.

Recall varies a lot across seeds (0.67–0.94). Which arms land in the test set matters. If your application cannot tolerate missing a third of collapsed chunks, do not deploy this alone.

How it was evaluated — split by ARM, not by frame

This matters more than the numbers. Three successive evaluations of this same task gave 1.000, +0.036 and +0.238, and only the last is real:

1. Random FRAME split → accuracy 1.000. Frames within a chunk are consecutive video frames, i.e. near-duplicates. A frame split puts copies of the same chunk on both sides and the model memorises chunks.
2. Split by CHUNK → lift +0.036. Better, but within an arm the collapse rate is near-constant (one arm is 98% collapsed, another 8%), so the model can learn "this looks like arm X" instead of learning collapse.
3. Split by ARM → lift +0.238. Entire generation runs held out. The test conditions were never seen in any form. This is the number above.

Labels

Ground truth comes from chunk_qc.inspect_chunk — an existing detector using SIFT keypoint count, Laplacian sharpness and face presence, probed at four points within each chunk. Labels are therefore that detector's opinion, not a human's. This model distils a ~100 ms multi-stage CV pipeline into a 0.131 ms forward pass; it inherits that detector's biases and cannot exceed it.

Input and the crop assumption

float32[N,3,64,64], RGB, scaled to [0,1], resized from a full frame.

It was trained on whole frames where collapse affects the entire image. It is not a crop-and-classify model. Nano vision models are known to be unlearnable below roughly 1% object-to-frame ratio against real backgrounds — so for any task where the target is small, crop to it first. That is not this model's regime; localised artifacts in an otherwise good frame are a documented failure mode below.

Known failure modes

• Localised damage. A frame with one corrupt region and an otherwise clean scene will likely pass. Global degradation is what it learned.
• Recall instability — see the seed spread above.
• Two generators only. LTX-Video-2B and Wan 2.2. Other models' failure signatures may differ.
• One scene type. Interior rooms, fixed camera. No outdoor, motion, or multi-subject footage.
• Inherits chunk_qc's blind spots, including its inability to detect mediocrity.

Learnability score

A training-free learnability ratio R (per loglens-learnability) scores this task at 0.337, the highest of eight tasks tested. However, that predictor failed validation as a general gate — across eight tasks it ranked them at Spearman ρ ≈ 0.57–0.62, so its score should not be read as evidence about this model. The measured lift above is the evidence. See NEGATIVE_generalised_R.md in the study repo.

Usage

import onnxruntime as ort, numpy as np, cv2
s = ort.InferenceSession("quality_gate.onnx", providers=["CPUExecutionProvider"])
img = cv2.resize(cv2.imread("frame.jpg"), (64, 64))          # BGR, full frame
x = img.astype(np.float32).transpose(2, 0, 1)[None] / 255.0
collapsed = int(s.run(None, {"image": x})[0].argmax(1)[0])   # 1 = collapsed

Latency (measured, Jetson AGX Orin, ONNX Runtime CPU)

mode	ms/frame	fps
1 thread	0.313	3,191
default threads	0.131	7,641
batch 32	0.061	16,513

Related

Part of the LogLens study. Tools: tripod, warpcost, vae-mtf, nullcheck.

MIT.