README.md

metadata

title: Flashback
emoji: 🔦
colorFrom: indigo
colorTo: gray
sdk: static
app_file: index.html
pinned: false
license: apache-2.0
short_description: Find the step your training run broke on, from 452 B/step
tags:
  - machine-learning
  - distributed-training
  - checkpointing
  - observability
  - debugging
  - anomaly-detection
  - pytorch

Flashback — find the step your training run broke on

Finding the step a training run broke on should cost kilobytes, not terabytes.

A large run fails. Somewhere in the last 40,000 steps a learning rate spiked, a bit flipped, a shard of the dataloader started serving garbage. Today you answer "when did this start?" by squinting at a loss curve and reloading whichever checkpoint happens to be nearest — so your answer is quantised to the checkpoint interval, and only exists at all if the loss actually moved.

Flashback keeps a 452-byte sketch of every single step — 128 internal statistics that usually move long before the loss does — and bisects that index with zero state reconstructions.

What this Space is

The telemetry here is real: recorded from actual training runs of a small Transformer in which a specific fault was injected at a specific step, with the ground truth written down before any detector looked at the data. The page reimplements flashback.bisect in JavaScript so it can run for free as a static Space, and checks itself against the answers Python computed for the same data — the verification badge at the bottom of the page says whether the port currently agrees.

Pick a fault, then compare the two charts. For the bit-flip scenario the loss curve never crosses its own anomaly threshold at any point in the run; the sketch crosses it on the exact step.

Measured results

12 runs across 6 fault types, ground truth recorded before any detector runs:

condition	mean error	exact hits	never detected
(A) full checkpoint every 100 steps + loss curve	20.0	0/12	4
(B) full checkpoint every step + loss curve (ideal)	0.5	4/12	4
(C) Flashback sketch + bisect	0.0	12/12	0

Add it to your own run

from flashback.integrations import FlashbackCallback

trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")])

flashback bisect my-run --metric grad_norm

Links

• Code: https://github.com/NagaYu/flashback
• Dataset: https://huggingface.co/datasets/NagaYu/flashback-forensics
• Detector model: https://huggingface.co/NagaYu/flashback-first-bad-step

What this demo is honest about

• The detector needs a healthy stretch to calibrate against. A run broken from step 0 has no baseline and will not be localised.
• It detects regime changes, so a fault that ramps in smoothly over hundreds of steps has no single correct answer. The data poisoning scenario here is off by ~36 steps at this model size, and the page shows that rather than hiding it.
• No bitwise determinism is claimed. This is a statistical detector over recorded statistics.
• The demo model is tiny (0.13M parameters, 600 steps) so the page stays a few megabytes. The mechanism is size-independent; the repository's benchmark runs the same experiments up to ~30M parameters.
• A full Gradio version of this demo — which additionally generates fresh runs on demand — is in the repository as app.py. Hugging Face requires a PRO subscription to host Gradio Spaces, so the free public demo is this static one.