--- title: Flashback emoji: 🔦 colorFrom: indigo colorTo: gray sdk: static app_file: index.html pinned: false license: apache-2.0 short_description: Find the step your training run broke on, from 452 B/step tags: - machine-learning - distributed-training - checkpointing - observability - debugging - anomaly-detection - pytorch --- # Flashback — find the step your training run broke on **Finding the step a training run broke on should cost kilobytes, not terabytes.** A large run fails. Somewhere in the last 40,000 steps a learning rate spiked, a bit flipped, a shard of the dataloader started serving garbage. Today you answer *"when did this start?"* by squinting at a loss curve and reloading whichever checkpoint happens to be nearest — so your answer is quantised to the checkpoint interval, and only exists at all if the loss actually moved. Flashback keeps a **452-byte sketch of every single step** — 128 internal statistics that usually move long before the loss does — and bisects that index with **zero state reconstructions**. ## What this Space is The telemetry here is **real**: recorded from actual training runs of a small Transformer in which a specific fault was injected at a specific step, with the ground truth written down before any detector looked at the data. The page reimplements [`flashback.bisect`](https://github.com/NagaYu/flashback/blob/main/flashback/bisect.py) in JavaScript so it can run for free as a static Space, and **checks itself against the answers Python computed for the same data** — the verification badge at the bottom of the page says whether the port currently agrees. Pick a fault, then compare the two charts. For the bit-flip scenario the loss curve never crosses its own anomaly threshold at any point in the run; the sketch crosses it on the exact step. ## Measured results 12 runs across 6 fault types, ground truth recorded before any detector runs: | condition | mean error | exact hits | never detected | |---|---:|---:|---:| | (A) full checkpoint every 100 steps + loss curve | 20.0 | 0/12 | 4 | | (B) full checkpoint every step + loss curve (ideal) | 0.5 | 4/12 | 4 | | **(C) Flashback sketch + bisect** | **0.0** | **12/12** | **0** | ## Add it to your own run ```python from flashback.integrations import FlashbackCallback trainer = Trainer(..., callbacks=[FlashbackCallback("runs/my-run")]) ``` ```bash flashback bisect my-run --metric grad_norm ``` ## Links - **Code**: - **Dataset**: - **Detector model**: ## What this demo is honest about - The detector needs a **healthy stretch** to calibrate against. A run broken from step 0 has no baseline and will not be localised. - It detects **regime changes**, so a fault that ramps in smoothly over hundreds of steps has no single correct answer. The `data poisoning` scenario here is off by ~36 steps at this model size, and the page shows that rather than hiding it. - **No bitwise determinism is claimed.** This is a statistical detector over recorded statistics. - The demo model is tiny (0.13M parameters, 600 steps) so the page stays a few megabytes. The mechanism is size-independent; the repository's benchmark runs the same experiments up to ~30M parameters. - A full Gradio version of this demo — which additionally generates fresh runs on demand — is in the repository as `app.py`. Hugging Face requires a PRO subscription to host Gradio Spaces, so the free public demo is this static one.