- pi05-piper1h-ea β Οβ.β
finetuned on Piper H (three checkpoints)
- β οΈ Read this first
- Shared recipe
- Run A β
pi05_piper1h_ea(1-hour mixture, 2,400 steps) - Run B β
pi05_piper1h_teleop(teleop only, 600 steps) - Run C β
pi05_piper20m_ea(20-minute mixture, 800 steps) - Comparing the runs
- Data preparation β what we changed
- Learnings and takeaways
- Caveats and open questions
- Repo layout
- Usage
- License
- β οΈ Read this first
pi05-piper1h-ea β Οβ.β finetuned on Piper H (three checkpoints)
Οβ.β
(flow matching) finetuned from pi05_base for a 14-DoF Piper H robot, on real Piper H teleop
and retargeted EgoDex human video.
This repo started as a single run and now holds three checkpoints from three runs that share one recipe and differ only in what data they saw. They are kept together because they are only interpretable against each other.
| # | Run / config | Data | Draw | Steps | Checkpoint | Path in this repo | W&B |
|---|---|---|---|---|---|---|---|
| A | pi05_piper1h_ea |
1 h mixture: 20.9 min teleop + 40 min ego | 24 teleop + 40 ego | 2,400 | 2399 | repo root | 4ksr9cp0 |
| B | pi05_piper1h_teleop |
teleop only (same 20.9 min) | 64 teleop | 600 | 599 | piper1h_teleop/599/ |
blk1ni8c |
| C | pi05_piper20m_ea |
20 min mixture: 10 min teleop + 10 min ego | 32 teleop + 32 ego | 800 | 799 | piper20m_ea/799/ |
9wyobg5w |
All three are raw Orbax training checkpoints (params + train_state + norm stats), not converted HF-format models. They are meant to be loaded by openpi. ~13 GB each, ~39 GB total.
NOTE ON "BIMANUAL": the rig is bimanual and the tensors are 14-D, but action dims 7β13 are
identically zero in every dataset used here β min and max are exactly 0.0. This is a single-arm
capture (the right arm, matching the pick_and_lift_right task name) padded into a bimanual
container. Only 7 of the 32 model action dims carry any signal. Read every loss number below with
that in mind.
β οΈ Read this first
| Checkpoints | Final step only, for each of the three runs. No intermediates. |
| Evaluation | None, for any of the three. Every number in this card is training loss. |
| Holdouts | Withheld correctly and verified in the logs β but never scored. |
All three runs completed cleanly (EXIT=0). Because there is no evaluation, "final" means last,
not best, and nothing here establishes that any one of the three is better than the others.
The three losses are NOT directly comparable. This is the single most important thing on this page, and it is explained in Comparing the runs below. In short: they sit at different points on different cosine schedules, and β worse β each run normalizes its actions with its own quantiles, so the losses are not even measured on the same scale.
Shared recipe
Everything below is identical across A, B and C. Only the data and step count differ.
| Field | Value |
|---|---|
| Model | Pi0Config(pi05=True) β PaliGemma backbone + flow-matching action expert |
| PaliGemma variant | gemma_2b_lora |
| Init | gs://openpi-assets/checkpoints/pi05_base/params (12 GB) |
| Action dim | 32 (14 real, 18 zero-padded β and 7 of the 14 are themselves constant zero) |
| Action horizon | 30 (not the openpi default of 50 β see Why H=30) |
| Max token len | 200 |
| Batch size | 64 |
| Optimizer | AdamW, clip_gradient_norm=1.0 |
| LR schedule | Cosine, warmup β peak 3.5e-5 β decay 3.5e-6 over the full run |
| EMA | off (ema_decay=None) |
| Dataloader workers | 16 |
| Normalization | quantile (use_quantile_norm=True, q01/q99), recomputed per run |
| Augmentation | ImageAugment(brightness=0.15, contrast=0.15, saturation=0.1, hue=0.03, crop_scale=(0.95,1.0)), geometric crop on base_0_rgb only |
| Hardware | 2Γ A100-SXM4-80GB (NVLink), data-parallel, --fsdp-devices 1 |
| Throughput | ~3.0 s/step on all three runs |
| Env | XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 (required β batch 64 OOMs at the 0.75 default) |
| Code | angkul07/openpi, branch piper1h-pi05 (A: dd8fa13, B: 1946f26, C: a079613) |
The LR schedule was inherited unchanged from the YAM arms of this series so that the mixture and the embodiment stay the only moving variables. It was not tuned for Piper.
Why H=30 and not the default 50
Two independent reasons, either sufficient:
openpi's LeRobot path clamps
delta_timestampsat episode ends β it repeats the final action rather than dropping the sample β and nothing in openpi consumesis_pad. So clamped steps enter the flow-matching loss as genuine regression targets, not ignored padding. The clamped share of supervised action steps is(Hβ1)/2L. Per dataset:mean episode length clamped @ H=30 clamped @ H=50 teleop (1 h, 154 eps) 162.8 8.9% 15.1% ego (1 h, 613 eps) 78.2 18.5% 31.3% teleop (20 min, 75 eps) 164.7 8.8% 14.9% ego (20 min, 153 eps) 78.8 18.4% 31.1% This is signal corruption, not reweighting.
20 Hz. H=30 at 20 Hz is 1.5 s, the same physical horizon H=50 gave the 30 Hz YAM arms (1.67 s). Inheriting 50 here would have silently asked for a 2.5 s chunk.
Observation space (all three runs)
Two real cameras, not three. The rig stores three, but the key names do not describe their content β this was established by extracting frames and looking, not by reading metadata:
| dataset key | teleop content | ego content | model slot |
|---|---|---|---|
front |
top-down workspace | full egocentric frame | base_0_rgb β real |
right |
right-arm wrist camera | tight crop on grasp point | left_wrist_0_rgb β real |
top |
sideways view of the robot | wide crop on grasp point | dropped |
| β | (no camera) | (no camera) | right_wrist_0_rgb = zeros, masked off |
front and right genuinely correspond across the two halves. top cannot: a head-mounted ego
camera can never produce a sideways view of the robot, so that key holds two unrelated images under
one name.
It is dropped in the repack rather than masked, because Pi0.embed_prefix runs SigLIP on every
entry of obs.images regardless of the mask β masking costs the same compute while keeping the
contradictory content. The surviving wrist view goes in slot 1 with slot 2 as the masked padding
slot, matching droid_policy and libero_policy.
Note the teleop-only run (B) keeps this exact two-camera repack even though it has no ego half, so the observation space is identical across all three.
Caveat: dropping top at the repack does not skip its video decode β LeRobotDataset decodes
every video feature before transforms run.
Also: 14-D joint state fed as a discrete state token (TokenizePrompt(discrete_state_input=True),
the Οβ.β
convention), and a language prompt from the LeRobot task string (prompt_from_task=True).
Action space (all three runs)
- 14-D, horizon 30, zero-padded to 32.
DeltaActions(mask=(TrueΓ6, False, TrueΓ6, False))β the 12 arm-joint dims are predicted relative to current state; the 2 gripper dims stay absolute.- Verified empirically that both halves store absolute actions, which is what this mask assumes:
mean|action β state|is 0.005 (teleop) and 0.022 (ego) againstmean|action|of 0.73 and 1.16. - The mask is symmetric per arm (6 joints + 1 gripper, twice), so it is correct regardless of which arm occupies dims 0β6 β which matters, because the active arm is the right one.
- Zero-variance dims are safe: both normalize paths carry a
+ 1e-6guard, so the 25 constant dims produce no NaN.
Run A β pi05_piper1h_ea (1-hour mixture, 2,400 steps)
Checkpoint 2399, at the root of this repo. Wall clock 2 h 02 m 59 s for 2,400 steps.
| Source | Train frames | % frames | Per batch | Gradient share | Epochs |
|---|---|---|---|---|---|
| Piper H teleop (real) | 23,347 | 32.7% | 24 | 37.5% | 2.47 |
| EgoDex β Piper H (retargeted) | 47,953 | 67.3% | 40 | 62.5% | 2.00 |
| Total | 71,300 | 100 | 64 | 153,600 frames seen |
Teleop is 154 episodes / 25,075 frames before the holdout; 15 episodes (holdout_fraction=0.1,
seed 0 β indices 2, 5, 10, 25, 38, 44, 72, 76, 89, 92, 97, 119, 121, 137, 149) are withheld. Ego is
613 episodes / 47,953 frames, no holdout. Both LeRobot v2.1 at 20 fps.
Teleop's gradient share (37.5%) is 1.15Γ its storage share β a deliberate but mild oversample toward the real-robot half. The draw was chosen to put both sources at ~2 epochs; with pools 2.05Γ apart, a 32/32 draw would have reached ego 2.0 epochs only at 3,000 steps, by which point teleop is at 4.24.
LR warmup 60. Checkpoints every 250 (max_to_keep=2, keep_period=1000); 1000/2000/2250 were
written during the run and are not uploaded. Norm stats piper1h_p50.
| Step window | Loss | chunk_first | chunk_last | grad_norm | param_norm |
|---|---|---|---|---|---|
| 0 β 100 | 0.04832 | 0.03718 | 0.07005 | 0.20740 | 1803.16 |
| 100 β 250 | 0.02526 | 0.00909 | 0.04255 | 0.09605 | 1803.18 |
| 250 β 500 | 0.02232 | 0.00819 | 0.03783 | 0.09034 | 1803.21 |
| 500 β 1,000 | 0.01893 | 0.00666 | 0.03138 | 0.08734 | 1803.28 |
| 1,000 β 1,500 | 0.01540 | 0.00577 | 0.02495 | 0.08917 | 1803.33 |
| 1,500 β 2,000 | 0.01286 | 0.00519 | 0.02074 | 0.08621 | 1803.36 |
| 2,000 β 2,400 | 0.01170 | 0.00489 | 0.01844 | 0.08489 | 1803.36 |
First-step loss 0.19416; minimum single-step loss 0.00741 at step 2,182; param_norm drift
+0.2074 (+0.0115%). Gradient clipping bound on exactly 3 steps β 0, 1, 2 (1.170, 1.244,
1.052) and never again.
Run B β pi05_piper1h_teleop (teleop only, 600 steps)
Checkpoint 599, at piper1h_teleop/599/. Wall clock 34 m 28 s.
The single-source control for A. Same embodiment, same cameras, same horizon, same LR β the only variable removed is the ego half.
| Source | Train frames | Per batch | Epochs |
|---|---|---|---|
| Piper H teleop (real) | 23,347 | 64 | 1.64 |
Same 154-episode teleop dataset and the same 15-episode holdout as A (selection is deterministic
in total_episodes/fraction/seed, and the source dataset is unchanged, so these are literally the
same episodes). LR warmup 30. Norm stats piper1h_teleop_only β recomputed, and not optional; see
Comparing the runs.
This is less teleop exposure than A got. A drew 24 teleop for 2,400 steps = 57,600 teleop frames (2.47 epochs); B sees 38,400 (1.64 epochs), i.e. 0.67Γ, on 4Γ fewer optimizer steps. So B-vs-A is not a clean ablation of "ego added or not" β it also cuts teleop exposure and total steps. To isolate the ego contribution properly, a teleop-only arm needs 973 steps to match teleop frames, or 2,400 to match optimizer steps. 600 was chosen for cost.
| Step window | Loss | chunk_first | chunk_last | grad_norm | param_norm |
|---|---|---|---|---|---|
| 0 β 100 | 0.04236 | 0.03132 | 0.07582 | 0.18417 | 1803.16 |
| 100 β 250 | 0.02452 | 0.01071 | 0.04983 | 0.09861 | 1803.18 |
| 250 β 400 | 0.02000 | 0.00755 | 0.04015 | 0.08651 | 1803.20 |
| 400 β 600 | 0.01669 | 0.00677 | 0.03310 | 0.08965 | 1803.21 |
First-step loss 0.16931; minimum 0.01209 at step 504; param_norm drift +0.0493 (+0.0027%).
Clipping bound on steps 0 and 1 only (1.188, 1.252).
Run C β pi05_piper20m_ea (20-minute mixture, 800 steps)
Checkpoint 799, at piper20m_ea/799/. Wall clock 44 m 42 s.
A small, cheap mixture on the same recipe as A β roughly 1/3 the data and 1/3 the steps.
| Source | Train frames | % frames | Per batch | Gradient share | Epochs |
|---|---|---|---|---|---|
| Piper H teleop (real) | 11,035 | 47.8% | 32 | 50% | 2.32 |
| EgoDex β Piper H (retargeted) | 12,049 | 52.2% | 32 | 50% | 2.12 |
| Total | 23,084 | 100 | 64 | 51,200 frames seen |
Teleop is 75 episodes / 12,048 frames with 8 withheld (indices 1, 2, 5, 19, 22, 35, 43, 57);
ego is 153 episodes / 12,049 frames, no holdout. LR warmup 40. Norm stats piper20m_p50.
The draw is 32/32, not A's 24/40, and that is the same rule rather than a different one. A's 24/40 was derived from wanting ~2 epochs per source against 1:2 pools. C's pools are already 1:1 (12,048 vs 12,049 frames), so the same rule gives 32/32. C is therefore not a proportional scale-down of A: A is 1:2 teleop:ego on disk and 37.5/62.5 by gradient, C is 1:1 and 50/50. Anything read off C about mixture ratio does not transfer to A.
| Step window | Loss | chunk_first | chunk_last | grad_norm | param_norm |
|---|---|---|---|---|---|
| 0 β 100 | 0.04306 | 0.03204 | 0.06342 | 0.19434 | 1803.16 |
| 100 β 250 | 0.02208 | 0.00884 | 0.03722 | 0.09758 | 1803.18 |
| 250 β 400 | 0.01839 | 0.00760 | 0.03034 | 0.08793 | 1803.20 |
| 400 β 600 | 0.01523 | 0.00650 | 0.02526 | 0.09158 | 1803.22 |
| 600 β 800 | 0.01286 | 0.00631 | 0.02009 | 0.08412 | 1803.22 |
First-step loss 0.19165; minimum 0.00786 at step 773; param_norm drift +0.0656 (+0.0036%).
Clipping bound on steps 0 and 1 only (1.258, 1.224).
C's data needed no repair β unlike A and B's teleop half, all 75 parquet files were free of the
datasets 4.x "List" metadata and all 684 videos already started at pts 0.000000. This was
checked, not assumed.
Comparing the runs
Final 200-step trailing means, side by side:
| A 1 h mix (2,400) | B teleop (600) | C 20 min mix (800) | |
|---|---|---|---|
| flow_loss | 0.01142 | 0.01669 | 0.01286 |
| chunk_first | 0.00455 | 0.00677 | 0.00631 |
| chunk_last | 0.01788 | 0.03310 | 0.02009 |
| chunk_last / chunk_first | 3.93Γ | 4.89Γ | 3.18Γ |
| grad_norm | 0.08374 | 0.08965 | 0.08412 |
| param_norm drift | +0.0115% | +0.0027% | +0.0036% |
| clipped steps | 0, 1, 2 | 0, 1 | 0, 1 |
Do not read this table as a ranking. Three confounds, in descending order of severity:
- The losses are not on the same scale. Each run computes its own q01/q99 normalization over its own pool, and the flow-matching loss lives in that normalized space. B's teleop-only quantiles are 5β9Γ narrower than A's mixture quantiles (e.g. actions dim 3: B spans [β0.126, 0.251], A spans [β1.832, 1.552]). Under a tighter normalizer the same physical error produces a larger normalized error. So B's higher loss does not mean B fits its data worse β a good part of that gap is the denominator. C's stats land within ~10% of A's, so this confound is small for A-vs-C and large for anything involving B.
- Different points on different cosine schedules. A ran 2,400 steps, C 800, B 600. Each decays 3.5e-5 β 3.5e-6 over its own length, so the final windows are not comparable snapshots. A has had 3β4Γ more optimizer steps.
- Different data. Different pool sizes, different mixture ratios, different epoch counts.
What the table does support: all three optimized cleanly and near-identically in shape β grad_norm settles to ~0.085 by step 100 in every run, clipping binds only in the first 2β3 steps, and no run shows instability. The recipe is reproducible across three data configurations.
The holdouts overlap β and there is exactly one clean comparison
C's 75 teleop episodes are a strict subset of A/B's 154: all 75 match by exact content hash of
observation.state + action. Its 153 ego episodes likewise match into A's 613. So the two holdouts
interleave, and this was measured rather than assumed:
- C's 8 held-out episodes map to A/B indices 2, 4, 10, 39, 45, 72, 89, 118. Of those, 4 were in A and B's training set (A/B indices 4, 39, 45, 118). So scoring A or B on C's holdout is 50% contaminated.
- Conversely, 4 of A/B's 15 held-out episodes (25, 76, 97, 149) are in C's training set. So scoring C on A/B's holdout is 27% contaminated.
The clean protocol: score all three checkpoints on the 11 episodes held out by both β A/B indices 2, 5, 10, 38, 44, 72, 89, 92, 119, 121, 137. Every one of those was withheld from all three runs. That is the only comparison across these three checkpoints that is not contaminated, and it has not been run.
Data preparation β what we changed
Two defects in the teleop half blocked training outright for runs A and B. Both were fixed locally on the training box, not upstream, so a fresh pull of the 1-hour dataset will reproduce them. Ego was clean on both counts, and run C's data was clean on both counts (verified).
1. Parquet metadata: datasets 4.x "List" β 3.x "Sequence"
The teleop converter embedded HuggingFace feature metadata written by datasets β₯ 4.x, which renamed
Sequence to List. openpi pins datasets==3.6.0, whose Features.from_dict has no List type, so
load_dataset() raised before training began:
ValueError: Feature type 'List' not found. Available: ['Value', 'ClassLabel', ..., 'Sequence', ...]
Fixed by rewriting "_type": "List" β "Sequence" in the schema metadata of all 154 teleop
parquet files. Only the metadata string changed; every file was verified for identical row count and
bit-identical observation.state / action values before the swap. Ego carried no huggingface
metadata at all, so its types are inferred from the arrow schema and it was unaffected.
2. Video PTS rebase: a uniform one-frame timestamp offset
Every one of the 462 teleop videos (154 episodes Γ 3 cameras) started at pts 0.05 instead of
0.0 β exactly one frame at 20 Hz. All 613 ego videos started correctly at 0.0. With lerobot's
default tolerance_s=1e-4 this is a hard crash mid-training:
AssertionError: query timestamps violate tolerance (0.0500 > tolerance_s=0.0001)
queried: 0.0 loaded: 0.05
No frames were missing. Teleop frame counts matched parquet rows exactly (195/195, 188/188,
174/174), spacing was a uniform 0.05, and the last frame sat +0.05 past the last parquet timestamp.
The entire PTS timeline was shifted by one frame β the encoder started its clock at 1/fps, not 0.
Fixed losslessly with ffmpeg -c copy -bsf:v setts=PTS-STARTPTS (no re-encode). All 462 were then
verified: frame count preserved, first pts 0.0, last pts equal to the episode's last parquet
timestamp.
Why rebase rather than widen tolerance_s. Raising the tolerance to ~0.05 would have "worked"
with a one-line change, but it maps dataset frames 0 and 1 onto video frame 0, leaving every teleop
sample with a one-frame-stale image while ego stays correctly aligned. The two halves would then
disagree about image/action alignment β the exact class of silent inconsistency that quietly degrades
a co-trained policy.
Note for anyone repeating this: -itsoffset -0.05 looks equivalent and silently drops a frame
(194 vs 195, measured). Use setts.
3. No gripper rescale β measured as unnecessary
The YAM arms of this series needed a gripper range alignment. That is not the case here, and it was
checked rather than assumed. Both halves store strictly binary {0.0, 1.0} grippers with the same
polarity:
| start | mid | end | fraction at 1 | |
|---|---|---|---|---|
| teleop | 0.745 | 0.677 | 0.083 | 0.50 |
| ego | 0.472 | 0.466 | 0.411 | 0.44 |
Same encoding, no inversion, so no rescale was applied. But see Caveats β the ego gripper carries almost no temporal signal, which is a different and arguably worse problem than a scale mismatch.
What was deliberately not changed
- Ego retargeting error was not filtered. Mean IK error is 4.770 cm over the 613 clips, and it is
bimodal, not uniform: 0.701 cm where the wrist-orientation constraint is never pinned, versus
6.944 cm where it is pinned in >50% of frames. The cause is Piper's joint5 range (Β±1.22 rad) against
the Β±1.571 the source retarget assumed.
_PIPER1H_EGO_EXCLUDEis empty in all three runs. - No resolution matching between halves. Teleop is 640Γ480 portrait, ego is 224Γ224 square; the resize to 224 squashes teleop ~1.33Γ vertically.
- Teleop was not rescaled or resampled. It is the reference.
Learnings and takeaways
1. Absolute loss is barely interpretable here, because 25 of 32 action dims are constant zero.
18 are standard zero padding, where x_t = tΒ·noise makes the target u_t = noise = x_t/t a
closed-form function of the input β free to fit. On top of that, 7 more are the all-zero left arm.
Roughly 78% of the objective is trivially predictable. The headline loss falling ~0.19 β ~0.043 in the
first 100 steps is mostly the model learning to emit zeros, in all three runs.
2. Normalization scale is a confound between runs, not just a preprocessing detail. This is the
lesson B taught. Recomputing norm stats per pool is mandatory β reusing A's piper1h_p50 for a
teleop-only run would have squeezed teleop into Β±0.1 of the [β1,1] axis and shifted the gripper
midpoint. But the necessary fix has a consequence people forget: it changes the units of the loss.
Any cross-run loss table where the runs have different norm stats is comparing different quantities.
Report the normalizer ranges alongside the losses, or compare only in physical units after
denormalizing.
3. chunk_last is the metric that carries information; chunk_first is nearly free. At the end
of each run chunk_last is 3.2β4.9Γ chunk_first. Predicting the action at t=0 is close to trivial
when actions are deltas from current state. The 1.5 s-ahead prediction is where the difficulty lives.
Note the caveat, though: 9% (teleop) to 18% (ego) of supervised steps are clamped repeats of the final
action, and those land disproportionately in the late part of the chunk β so chunk_last is partly
measuring how well the model predicts "hold still at the end", which is easier than real motion.
4. All three runs barely moved the base model, and the evidence is unambiguous. param_norm
drifted +0.0115% (A), +0.0027% (B), +0.0036% (C). With LoRA, a 3.5e-5 peak and ~2 epochs, essentially
all capability is inherited from pi05_base. That was the stated intent, and it means none of these
runs is likely to have overfit. It equally means you should not expect strong Piper-specific
specialisation from any of them. If eval comes back bland rather than wrong, under-training is the
first hypothesis, and more steps or a higher peak LR is the lever β not more data.
5. No run was length-limited. Loss fell monotonically in every window of every run with no
plateau, and chunk_last was still descending at the final step in all three. Step counts were chosen
to hit ~2 epochs per source, not because a curve flattened.
6. The recipe transfers across data scales without retuning. A, B and C differ by 3Γ in data and 4Γ in steps, and all three produced the same optimization signature: grad_norm ~0.085 from step 100, clipping only in the opening 2β3 steps, monotone descent, no spikes. The warmup fraction was the only knob that needed rescaling (2.5% at 2,400 steps, 5% at 600β800, because 2.5% of 600 is a 15-step ramp to peak). Whatever else is uncertain here, the optimizer settings are not the fragile part.
7. Subset datasets silently break holdout hygiene. C was built as a small cut of the same source
data as A and B, which is sensible β but it re-derives its holdout from total_episodes=75 instead of
154, so the two splits interleave rather than nest. The result is 50% contamination in one direction
and 27% in the other, which would have quietly inflated any cross-run eval. If you cut a subset of a
dataset you already have runs on, carry the parent's holdout episode IDs into the subset instead of
re-deriving the split. The 11-episode intersection is the workaround here; carrying IDs is the fix.
8. Aggregate loss cannot answer what these mixtures were built to ask. A single scalar over a
mixture cannot distinguish learning from the teleop half, the ego half, or fitting the 25 constant
dims. Per-source (loss_teleop / loss_ego) and real-vs-padded-dim splits remain the highest-value
cheap change: split jnp.square(v_t β u_t) before the final mean and carry a source_index through
MixtureDataset into the train step.
9. Check what the camera keys actually contain before mapping them. The teleop rig's names are
actively misleading β front is the top-down view, top is a sideways view of the robot, and only
right is honest. One frame extracted per key catches it; metadata does not.
10. Infrastructure notes worth reusing.
- 2Γ A100 SXM4 uncapped at 400 W ran 3.07 s/step vs 3.528 for an otherwise identical pair capped at
275 W β ~14% throughput for free. Check
nvidia-smi -q -d POWERbefore benchmarking a rented box. nprocreported 128 on this container; the real quota from/sys/fs/cgroup/cpu.maxwas 30.7 cores. Size worker pools off cgroup, nevernproc.num_workers=16is a non-lever β the loader uses ~3 cores and the run is compute-bound.XLA_PYTHON_CLIENT_MEM_FRACTION=0.9is required for batch 64 on 80 GB; the 0.75 default OOMs.- Οβ.β checkpoints measured 13 GB (6.7 GB params + 5.9 GB train_state), not the 16β17 GB budgeted.
- A dataset authored with
datasetsβ₯ 4.x will not load under openpi's pinned 3.6.0 if the converter embedded HF feature metadata. Cheapest check before renting a GPU:pq.read_schema(f).metadata. - Setting
save_intervalequal tonum_train_stepsyields exactly one checkpoint: a run of N steps covers indices 0β¦Nβ1, so the interval never fires mid-run and only the final write happens.
Caveats and open questions
These bound what the numbers above can support.
- No evaluation of any kind, for any checkpoint. Holdouts exist and were correctly withheld (verified in the training logs), but none was ever scored. Loss curves only.
- The data is single-arm in a bimanual container. Dims 7β13 are exactly zero, so the model learns 7 real dims out of 32. Any claim about "bimanual" capability is unsupported.
- The ego gripper channel is close to noise, and ego is 50β62.5% of the gradient in A and C. Teleop shows clean pick semantics (open 0.745 β closed 0.083); ego is flat at ~0.44 throughout with only a slight downward drift. Under Οβ.β 's few-step flow matching, a near-random gripper target on half or more of the samples is precisely the setup that mode-averages into a half-open hand. This is the first thing to suspect if grasping underperforms, and the fix is upstream in the ego gripper labels, not in hyperparameters.
- Most of the gradient in A and C comes from synthesized, retargeted data whose IK residual averages 4.770 cm. The sim-to-real gap of that share is unmeasured.
- Language conditioning comes almost entirely from the ego half. Teleop has 1 task string
(
pick_and_lift_right); the 1 h ego half has 100 and the 20 min ego half has 79.prompt_from_task=Truetherefore conditions the ego half and does essentially nothing for the half an eval would score. Run B, being teleop-only, is effectively unconditioned. - B is not a clean ablation of A. It cuts teleop exposure (1.64 vs 2.47 epochs) and optimizer steps (600 vs 2,400) at the same time as removing ego.
- C is not a scale-down of A. Different mixture ratio on disk (1:1 vs 1:2) and different draw (32/32 vs 24/40).
- Both data fixes were applied to the local copy of the 1-hour dataset, not upstream. Pulling it
fresh gives back the
Listmetadata and the 0.05 s PTS offset. - Single seed per configuration, no repeats. Nothing here isolates seed variance, and with three runs differing in several variables at once, no single-variable conclusion is available.
Repo layout
# Run A β 1-hour mixture, step 2399, at the repo ROOT
_CHECKPOINT_METADATA
params/ # Orbax OCDBT β inference weights (6.7 GB)
train_state/ # optimizer state β resuming only (5.9 GB)
assets/piper1h_p50/norm_stats.json
train_metrics_full.log # per-step metrics, steps 0β2399
train_console.log
# Run B β teleop only, step 599
piper1h_teleop/
599/
_CHECKPOINT_METADATA
params/
train_state/
assets/piper1h_teleop_only/norm_stats.json
train_metrics_full.log # per-step metrics, steps 0β599
train_console.log
# Run C β 20-minute mixture, step 799
piper20m_ea/
799/
_CHECKPOINT_METADATA
params/
train_state/
assets/piper20m_p50/norm_stats.json
train_metrics_full.log # per-step metrics, steps 0β799
train_console.log
Run A's checkpoint is at the repo root, not under a 2399/ directory. That is a historical
artifact of how it was first uploaded (upload_large_folder takes no path_in_repo). It is left as
is; B and C are namespaced. Adjust paths accordingly β this is the one asymmetry in the layout.
For inference you only need params/ and assets/ β train_state/ is ~46% of the bytes and is
needed only to resume training.
Usage
# Run A (1-hour mixture) β files live at the repo root
hf download angkul07/pi05-piper1h-ea \
--include "params/*" "assets/*" "_CHECKPOINT_METADATA" \
--local-dir ./ckpt_A
# Run B (teleop only)
hf download angkul07/pi05-piper1h-ea \
--include "piper1h_teleop/599/params/*" "piper1h_teleop/599/assets/*" \
"piper1h_teleop/599/_CHECKPOINT_METADATA" \
--local-dir ./ckpt_B
# Run C (20-minute mixture)
hf download angkul07/pi05-piper1h-ea \
--include "piper20m_ea/799/params/*" "piper20m_ea/799/assets/*" \
"piper20m_ea/799/_CHECKPOINT_METADATA" \
--local-dir ./ckpt_C
from openpi.policies import policy_config
from openpi.training import config as _config
# pick the matching config name for the checkpoint you downloaded
cfg = _config.get_config("pi05_piper1h_ea") # A
# cfg = _config.get_config("pi05_piper1h_teleop") # B
# cfg = _config.get_config("pi05_piper20m_ea") # C
policy = policy_config.create_trained_policy(cfg, "./ckpt_A")
# B: "./ckpt_B/piper1h_teleop/599"
# C: "./ckpt_C/piper20m_ea/799"
action_chunk = policy.infer({
"observation/front_image": front_rgb, # HxWx3 uint8 β top-down / egocentric view
"observation/right_image": right_rgb, # HxWx3 uint8 β wrist / grasp-point view
"observation/state": state_14d,
"prompt": "pick_and_lift_right",
})["actions"] # (30, 14)
The config must match the checkpoint. Each run shipped its own norm stats under its own asset id, and loading a checkpoint under the wrong config will silently apply the wrong normalization β the failure mode is plausible-looking but mis-scaled actions, not an error.
Note the observation keys: two images, not three. observation/top_image is not consumed β
PiperInputs reads only front and right, and fills right_wrist_0_rgb with a masked zero array.
Expect dims 7β13 of the returned actions to be ~zero, since they were constant in training.
Requires branch piper1h-pi05 of angkul07/openpi (revision a079613 has all three configs), along
with src/openpi/policies/piper_policy.py.
To resume training, point openpi at the parent directory:
uv run scripts/train.py pi05_piper20m_ea --exp-name piper20m_ea_v1 --fsdp-devices 1 --resume
License
Apache-2.0, inherited from openpi and pi05_base. The underlying datasets carry their own terms; the
EgoDex-derived half inherits EgoDex's license.