mv-sam3d-6d — Multi-View 6D Object Pose Flow Model

mv-sam3d-6d is a multi-view 6D object pose model built on top of the SAM-3D shape/decoder stack. Given multiple calibrated camera views of an object, it predicts the object's canonical shape (voxel), 6-DoF rotation, translation, and scale by plain flow matching in an anchor-frame canonical formulation. The repository publishes the full training checkpoint history for several runs, plus the migrator/ package needed to reproduce data preparation and training.

Checkpoint families

Checkpoints live under checkpoints/<family>/ and are named to encode the training step, the data they were trained on, and the SSI mode:

step_{step:06d}__{trained-data}__{ssi-mode}.pt
step_{step:06d}__{trained-data}__{ssi-mode}_optimizer.pt   (optimizer sidecar, 10k steps only)
step_{step:06d}__{trained-data}__{ssi-mode}.pt.metadata.json

Family	Path	Trained on	ssi_mode	Steps
DexYCB base	checkpoints/dexycb-scene-ssi/	DexYCB	scene	10k – 100k (every 2k)
Integrated scene-SSI	checkpoints/integrated-scene-ssi/	integrated 10 datasets	scene	8k – 80k (every 2k)
Integrated object-SSI	checkpoints/integrated-object-ssi/	integrated 10 datasets	object_centric	8k – 80k (every 2k)
DexYCB rot-aug	checkpoints/dexycb-aug-scratch/	DexYCB + rotation augmentation	scene	8k – 34k (every 2k)

The DexYCB-base run is the originally-evaluated model; its step-60k checkpoint (checkpoints/dexycb-scene-ssi/step_060000__dexycb__scene-ssi.pt) is the reference model reported in evaluation.

Integrated dataset list

The integrated runs are trained on 10 datasets: dexycb, pace, graspnet, ycbv, hocap, h2o, hograspnet, dexh2r, contactpose (held-out for evaluation: housecat6d, graspnet-novel, ho3d).

ssi_mode meaning

ssi_mode selects the shape-space input (SSI) normalization frame used during training and inference:

• scene — shape supervision / conditioning is expressed in the scene frame.
• object_centric — shape supervision / conditioning is expressed in a per-object canonical frame.

The two integrated runs are identical except for this switch, so they can be compared head-to-head.

Optimizers

Optimizer state is provided so training can be resumed:

• For the integrated and rot-aug runs, a separate ..._optimizer.pt sidecar is uploaded only at 10k-multiple steps (10k, 20k, 30k, …).
• For the DexYCB-base run, some checkpoints bundle the optimizer state directly inside the .pt file; these are uploaded as-is.

Each checkpoint's *.metadata.json records optimizer_included, optimizer_file, ssi_mode, trained_on, and (where available) validation metrics (shape_iou, rot_deg, trans_cm) at that step.

migrator/

The migrator/ folder is the data + weights migration package: it contains the SAM-3D shape encoder/decoder/generator checkpoints, the DINOv2 backbone, canonical object caches, and the scripts used to download/prepare datasets and launch training. Start there to reproduce training or to run inference with the checkpoints above.

Usage

Download a single checkpoint (example: the reference 60k DexYCB model):

from huggingface_hub import hf_hub_download
path = hf_hub_download(
    "Ronaldo-GOAT/mv-sam3d-6d",
    "checkpoints/dexycb-scene-ssi/step_060000__dexycb__scene-ssi.pt",
)