STATE.md

AgentPTB run state

Scope and compliance

• Objective: improve Qwen/Qwen3.5-9B-Base for agentic software engineering and shell work using weights trained in this run and/or a compliant pi harness.
• Evaluation-only data: /root/work/shared/tasksets/terminal-bench-2 and /root/work/shared/tasksets/swe-bench-verified. Task texts and solutions will never enter training corpora or selection/filtering criteria.
• Forbidden materials: other cells/runs, operator notes, benchmark repository, environment credentials as a model/data source, and third-party post-trained weights for initialization/merging.
• Permitted sources planned: the base checkpoint, public generic code/agent datasets, local synthetic tasks not conditioned on evaluation items, and trajectories generated by locally run public models if useful.

Environment

• Started: 2026-08-14 16:27 UTC
• Deadline: 2026-08-18 20:27 UTC
• GPUs: CUDA devices 4,5,6,7 (four NVIDIA B200 183 GB)
• Workspace: /mnt/pvc/users/simon/agentptb/runs/sol-max-s1/workspace
• Base: Qwen/Qwen3.5-9B-Base in the pre-staged Hugging Face cache
• Trainer: /root/work/a/prime-rl
• Important serving flags: --enable-auto-tool-choice --tool-call-parser qwen3_coder
• Important trainer setting: model.attn = "flash_attention_2"
• Important runtime settings: broker, block_network = false, exported PRIME_API_KEY

Experiment log

• Base inference server: configs/inference-base.toml, one B200 on physical GPU 4, port 8200. Correctly emits a structured bash call on a manual smoke. Required Qwen parser and multimodal limits confirmed in the resolved vLLM arguments.
• Base Terminal-Bench 2 smoke attempt: four fixed-seed shuffled tasks, evals/base-tb2-smoke-run. All four were invalid infrastructure results (zero model calls): two broker ready-timeout 408s and two HTTP read timeouts after first-time image preparation. Never count these as reward outcomes. The valid retry in evals/base-tb2-smoke-retry scored 1/4; this is much too small for a useful rate. One failure exhausted context and the success exhausted the episode output-token cap.
• Public SFT source: Kwai-Klear/SWE-smith-mini_swe_agent_plus-trajectories-66k, revision 750b2c11239fd5e32f97e6cfb9bf80fb9a9a2983, MIT. The published corpus contains 65,994 successful trajectories from synthetic SWE-smith issues; its reported Qwen3-8B SFT result is 39% SWE-bench Verified.
• Converted corpus: data/pi-swe-smith-66k (4.5 GB). Mechanical conversion only: fenced shell actions to Pi bash function calls, observations to tool messages, source terminal marker to a concise final assistant response. 65,994 rows, 2,197,438 tool calls, 147 malformed/plain actions, and 0 exact identifier overlaps with the local SWE-bench Verified task directory. Provenance is in the dataset directory.
• Public terminal SFT source: nvidia/Nemotron-Terminal-Corpus, revision a1667c4ffdadea02a89bffe4f1bb7ca2ff19f8d9, CC-BY-4.0. Used only its three broad, novel skill-based synthetic configs, not benchmark tasks or oracle solutions. Converted 131,564 rows and 796,049 shell-action batches to structured Pi calls in data/pi-nemotron-terminal-skill-140k; 8,277 malformed or special-key trajectories were rejected and zero exact evaluation instruction/name overlaps were found. Converter: scripts/prepare_nemotron_terminal_sft.py.
• A follow-up conservative exact normalized 14-token audit against both evaluation instruction sets removed 5,701 SWE rows and 3,835 terminal-skill rows. Clean components are data/pi-swe-smith-66k-clean14 (60,293 rows) and data/pi-nemotron-terminal-skill-140k-clean14 (127,729 rows). The active balanced mixture is data/pi-agent-mix-clean-v1: 127,729 terminal rows plus two deterministic SWE copies, 248,315 rows total, shuffled with seed 20260814.
• The initial unfiltered production attempt was stopped and discarded after step 17 as soon as this conservative audit completed. Its data mixtures and output are renamed with quarantined-/discarded- prefixes and cannot be selected accidentally.
• Optional broad adapter data is staged but is not part of the active run: the public Terminal-Corpus adapter split was converted to 196,767 Pi trajectories and 1,000,521 shell calls in data/pi-nemotron-adapters-226k-v1. A conservative exact normalized 14-token blacklist against both evaluation instruction sets removed 7,434 rows; interactive-key and actionless rows were also excluded. The fully clean fallback mixture is data/pi-agent-mix-clean-v2-adapters (445,082 rows).
• Full-parameter trainer smoke at 2K was stopped before an update because the window contained no assistant tokens and the loader cycled. Preserved under outputs/sft-smoke.
• Full-parameter trainer smoke at 8K completed one update successfully under outputs/sft-smoke-8k: loss 0.1684, grad norm 4.8696, peak 168.5/178.4 GiB on one B200. This validates data/rendering/model/backprop but shows full-parameter single-GPU memory is too tight for longer context.
• Rank-64 LoRA at 32K succeeded both with Ulysses CP=2 (28.8 GiB/GPU, 2m02s compile step) and CP=1 (48.0 GiB, 6.8k tokens/s steady state). CP is unnecessary on B200.
• Four-GPU full-parameter 32K training is feasible and preferable to LoRA: micro-batch 1 reached 7.3k aggregate tokens/s at 45.6 GiB/GPU; micro-batch 2 reached 11.6k at 48.1 GiB; micro-batch 4 reached 15.5k at 70.1 GiB; micro-batch 8 reached 21.3k at 115.2 GiB. All losses and gradient norms were finite. Production uses micro-batch 8.
• The clean production recipe follows the published Terminal-Corpus setup where practical: 32K, AdamW at 2e-5, weight decay 1e-4, beta2 0.95, cosine schedule with 10% warmup, global batch 128, full-parameter training. Config: configs/sft-agent-mix-clean-full-v1.toml; output: outputs/sft-agent-mix-clean-full-v1.
• Clean production started at 2026-08-14 18:17 UTC. Checkpoint 50 completed cleanly: loss 0.2909, recent mean 0.2697, no NaNs, 102 GiB resumable state, 18 GiB stable HF weights, and 143.8 seconds of save overhead. Step 51 then completed normally (loss 0.2772, grad norm 0.2748). Step 150 is projected near 2026-08-15 01:00 UTC including the two remaining checkpoint saves.
• Paired evaluation tooling is ready: stock and custom runs use the same fixed seed-0 sample, scripts/run_paired_evals.sh launches isolated concurrent runs, and scripts/compare_evals.py reports paired outcomes.
• Evaluation reporting now uses the conservative score and Wilson interval over every scheduled episode. Reward-bearing-only scores are explicitly diagnostic, and any transient errored episodes will be resumed on the same fixed sample rather than dropped from the denominator.
• The active loader advances about 243 raw trajectories per optimizer step. At that measured rate, the prepared step-2,050 continuation lands at about 2.01 epochs, matching Terminal-Corpus's published two-epoch recipe. The first 36,500 shuffled rows retain the full mixture ratio (51.44% terminal / 48.56% SWE).
• A structural quality audit found that 54,322/127,729 clean terminal rows end in an explicit non-tool assistant completion; the remaining source rollouts stop at a tool observation or turn limit. All 60,293 clean SWE rows finish explicitly. The independently filtered, evaluation-content-blind continuation mixture data/pi-agent-complete-mix-v1 contains two copies of each complete component (229,230 rows total). It is staged only; step 150 will be evaluated before any branch is trained on it.
• The public Terminal-Corpus paper's own broad ablation strongly favors retaining incomplete synthetic trajectories: 12.4% TB2 unfiltered versus 6.74% complete-only and 5.06% success-only. It also favors single-stage mixed training (13.03%) over a two-stage curriculum (10.39%). Therefore the evidence-backed main continuation remains pi-agent-mix-clean-v1; the complete-only corpus is only a possible narrow stopping ablation.
• Custom Pi now optionally advertises the true model context window and per-call generation ceiling. Custom configs set 65,536/4,096, enabling Pi's generic automatic compaction before a backend overlong-prompt error. Defaults remain unset, so stock Pi behavior is unchanged. A no-review custom variant is ready for a paired step-150 ablation of the generic review pass.
• Custom evaluation variants now accept Pi's clean empty end-turn after its tool activity, so a modified workspace reaches the task grader instead of becoming an unscored harness error merely because no separate prose was emitted. Stock Pi remains unchanged. The option parsed successfully in the pinned run environment; an attempted broad Verifiers test collection was blocked before tests by an unrelated installed prime_sandboxes API mismatch.
• Evaluation summaries now derive their conservative denominator from the saved num_tasks * num_rollouts configuration and explicitly count missing trace records. A client crash therefore cannot inflate the reported score by silently shrinking the denominator.

Current plan

Before any step-150 selection result was available, the continuation's purely mechanical benchmark set was extended with micro-batch 10/global-batch 120 and micro-batch 12/global-batch 96. The measured micro-batch 4-to-8 memory curve predicts both may fit on B200, and reducing accumulation could increase useful tokens per second. Each variant must still complete finite steps, remain below 170 GiB/GPU, save both resumable and HF step-152 checkpoints, and beat the same-batch-128 baseline throughput by at least 3%. The fastest eligible recipe wins. If a different global batch is selected, the deadline calculation uses its actual tokens/update and the terminal-step cap preserves the original batch-128 two-epoch token budget. This changes no data, score gate, or model branch.

The waiting agentptb-post-step150 session was restarted after this edit so its shell reads the new benchmark set. At restart the evaluation-complete marker did not exist; the live step-150 evaluator and inference process were left untouched. All six benchmark and production TOMLs parse, both new global batches satisfy Prime-RL's divisibility invariant, and the handoff script passes bash -n.

At 2026-08-14 21:04 UTC, a focused Prime-RL unit-test invocation triggered the suite's module-level pkill -f torchrun cleanup fixture and terminated the live trainer after step 63. This was a test-isolation error, not a model or data failure. Steps 51--63 were not checkpointed and were discarded.

The first resume exposed a Prime-RL CPU-offload edge case: optimizer tensor states and the scheduler counter loaded, but optimizer param-group LR did not, pinning steps 51--52 near 2e-13. That attempt was stopped and discarded. The documented workaround is now in the same resume config: skip_scheduler = true, a new 100-step cosine with no warmup, and peak LR 1.7e-5 (the actual LR at checkpoint 50). Model, optimizer moments, progress, and dataloader still load from step 50.

Active trainer tmux: agentptb-sft-clean-resume50b. At 2026-08-14 21:18 UTC it started the corrected resume; step 51 completed at 21:22 with loss 0.2772, gradient norm 0.2748, and the intended LR 1.70e-5. Do not run the root Prime-RL pytest suite alongside training unless PRIME_RL_TEST_SKIP_PROCESS_CLEANUP=1 is explicitly set. The metrics file keeps stale rows 51--63 from the killed attempt and 51--52 from the near-zero-LR attempt; later appended rows supersede them, while training_status.py will continue to show 63 until the corrected resume passes it. At 21:42 UTC the corrected run reached step 59 with finite loss 0.2824, gradient norm 0.2982, LR 1.67e-5, and 28.0k tokens/s; all corrected-resume updates through that point were healthy.

At 21:50 UTC the corrected run reached step 62 with loss 0.2786, finite gradient norm 0.2788, intended LR 1.65e-5, and 28.0k tokens/s. A deterministic 2,000-row structural sample of the active mixture found an exact SFT system prompt match in every row, only the aligned bash structured tool in assistant actions, a median/p95 of 11/51 assistant turns, and an explicit final assistant completion in 70.0% of rows. Every intermediate assistant tool turn is still rendered with its own supervised end-turn token. This supports the 64-turn custom ceiling and does not justify switching to the staged completion-only branch before seeing actual candidate stop behavior.

A direct step-50/base tensor comparison confirms lineage and nonzero updates: model.language_model.layers.0.input_layernorm.weight differs from base by mean/max absolute 7.71e-5/4.88e-4, while the frozen model.visual.blocks.0.norm1.weight is exactly identical to base.

Separate evenly spaced component samples also checked the row-level balance. Terminal-skill rows average 6.47 assistant turns and about 21.3k assistant characters; SWE-smith rows average 33.96 turns but about 25.6k assistant characters. Because the mixture has 127,729 terminal rows and 120,586 repeated SWE rows, supervised assistant volume is therefore close to balanced rather than being dominated by SWE's longer tool-output histories. The public Terminal-Corpus ablation confirms that adding all adapter sources gives only a small TB2 mean gain over skill-only data (13.03% versus 12.4%) when each recipe is trained for two full epochs. At this run's fixed remaining compute, the 445k-row adapter-expanded mixture would receive under one epoch and dilute the high-value SWE source. Keep the evidence-backed 248k-row active mixture for the long continuation unless candidate evaluation reveals a clear contrary failure.

All staged resumed-training configs now use zero-warmup cosine restarts. With CPU-offloaded optimizer state, constructing a warmup scheduler before checkpoint load mutates the param-group LR to the warmup floor, and loading tensor state does not restore that group. Fresh-from-base configs retain their normal warmup.

The guarded agentptb-post-step150 watcher now prevents an idle gap after the step-150 evaluations. Once evaluation has completed and inference has fully released the GPUs, it hard-links the resumable step-150 state into an isolated two-update selective-activation-checkpoint benchmark. The benchmark now must successfully save both the resumable and HF-compatible step-152 checkpoints, which exercises the temporary GPU staging of CPU-offloaded Adam state that its earlier train-step-only check missed. It selects selective AC only after that save, at >=29.0k tokens/s and <=170 GiB peak training memory; otherwise it uses the proven full-AC config. It then launches the same-mixture step-1,800 continuation as agentptb-sft-continuation. Decisions and the measured benchmark are written under state/; no dataset branch is selected automatically.

The handoff now also benchmarks full activation checkpointing with GPU-resident Adam state, plus full checkpointing with activation offloading disabled while Adam stays on CPU. Four-way B200 sharding may fit either despite the source paper's H100-oriented offload recipe, and avoiding one class of host traffic could improve throughput. The proven recipe and all three experimental variants independently resume step 150, perform steps 151--152 on the same dataloader state, and must show finite metrics, LR in [1.9e-5, 2.1e-5], no more than 170 GiB peak training memory, and successful resumable plus HF checkpoint saves. Because packed-document lengths make absolute throughput batch-dependent, an experimental variant must beat the paired proven-recipe baseline by at least 3%; the fastest eligible variant wins, otherwise the baseline recipe remains the fallback. The running post-step150 watcher was restarted after this change. Benchmark and inference launchers now execute through a setsid bash wrapper, and cleanup probes and kills the whole process group even when a uv leader exits before torchrun workers. The sleeping step-150 evaluation and post-evaluation watchers were restarted to load this cleanup fix; the live trainer was untouched. The initial continuous target was step 1,800 (about 1.74 corpus epochs). At the proven fallback throughput, including the three save-through benchmarks, this still leaves roughly 20 hours before the deadline for checkpoint checks and the full 1,178-episode canonical stock/custom measurement. The chosen resume file's historical max_steps = 1650 is overridden explicitly on the launch command and recorded in state/continuation-config.txt; the resolved trainer config will therefore preserve the actual 1,800-step schedule. The initial deadline guard fell back to step 1,650 if fewer than 87 hours remain after development evaluation and the three mechanical benchmarks. The preservation watcher reads the recorded resolved target, so either terminal checkpoint is retained and labelled correctly. At the expected handoff time the 1,800-step branch had about two hours of additional guard beyond this threshold. This discrete rule was superseded before evaluation by the score-independent throughput/time calculation recorded at the end of this section.

All step-150 configs now use one typed whole-episode retry limited to explicit sandbox/network infrastructure exceptions; context, model, harness, and task failures remain single-shot. Dry config validation confirmed the retry policy, and the temporary dry-run output directories were removed before evaluation. True stock configs leave every newly added Pi option unset (including text_only), while custom configs retain the aligned prompt and other opt-ins.

Canonical final configs are staged for all 89 TB2 and all 500 SWE-bench Verified tasks. The four-run stock/custom launch allocates 16 concurrent episodes to each TB2 run and 48 to each SWE run, totaling the measured-safe 128 sandbox ceiling. scripts/run_final_evals.sh requires an explicit review/no-review/stock choice plus sampling temperature and refuses to overwrite existing results. The wrapper consumes both precommitted state files. The selected model will be exposed through submission/model only after checkpoint selection.

At 22:36 UTC a takeover audit caught a completion-sentinel mismatch in the post-step-150 benchmark watcher before it ran. Prime-RL resumable checkpoints finish with checkpoints/step_N/trainer/.metadata; only HF weight exports have a root STABLE marker. The watcher had required a nonexistent resumable STABLE, which would have falsely rejected all four execution benchmarks. Both the early-return and polling predicates now require the real metadata file plus the HF STABLE marker, bash -n passes, and only the sleeping agentptb-post-step150 session was restarted. The active trainer was untouched. At the same audit the clean resume was at step 80 with finite loss 0.2646, gradient norm 0.2875, intended LR 1.37e-5, 27.6k tokens/s recent throughput, and 116.2 GiB/GPU peak memory. The SFT rows' system prompt and four-tool schema were also checked directly against the custom Pi prompt/schema.

The final sampling and review choices are fixed before candidate results are visible. Qwen's public Qwen3.5 model card recommends temperature 0.6/top-p 0.95 for thinking-mode precise coding, whereas the initially staged temperature 0.2 had not been measured. The development gate therefore evaluates aligned no-review and review at both temperatures on the same 32+32 tasks. Temperature 0.6 is the evidence-backed default; 0.2 replaces it only if it solves at least as many tasks on each suite, has a strictly positive combined paired advantage, and its combined two-sided exact sign-test p-value is at most 0.10. At the selected temperature, review is enabled under the identical non-inferiority, positive-advantage, and p-value rule; otherwise use aligned no-review. Missing or errored episodes count as failures, and reward-bearing-only rates do not select either setting. The SFT-aligned scaffold remains the default; stock replaces it only under the identical per-suite non-inferiority, positive paired advantage, and p-value gate. Ten primary development runs at concurrency 12 each keep total sandbox concurrency at 120, followed by two stock t=0.6 runs at concurrency 32 each (64 total). scripts/select_harness.py implements this rule, and the sleeping step-150 evaluation watcher writes the full decision JSON, selected temperature, and selected review mode under state/ before releasing the continuation watcher.

Final checkpoint selection is likewise score-independent. Select the recorded terminal target if its HF export has STABLE, every indexed shard exists and loads with the declared shapes, the terminal training metrics and gradients are finite, and a served structured-tool-call smoke succeeds. If and only if that artifact is corrupt or training becomes numerically unhealthy, walk backward through the scheduled stable checkpoints and take the newest one satisfying the same health checks. No intermediate checkpoint is benchmarked to make this choice.

Checkpoint 100 completed and passed the independent header/metric gate at 23:28 UTC. The resumable trainer/.metadata and HF STABLE markers exist; all four shards expose exactly 760 indexed BF16 tensors totaling 9,409,813,744 elements. Step-100 loss was 0.2561, gradient norm 0.2770, LR 8.77e-6, and NaN count zero. The trainer continued normally toward step 150.

agentptb-finalize-when-ready now closes the last idle gap. It reads the recorded continuation target, waits for the preservation watcher to hard-link that terminal HF export and for its tmux session to exit, then runs the full finite-tensor/package gate and the process-group-owned final inference/evaluation pipeline. Any health or pipeline failure stops closed and leaves logs/results for manual recovery; it never chooses a checkpoint from scores.

At 23:53 UTC the active resume reached step 110 with loss 0.2685, recent-mean loss 0.2612, gradient norm 0.2471, zero NaNs, and a 150.0-second recent mean update time. Step 150 was projected about 1.7 hours later. All five trainer/evaluation/continuation/preservation/finalization tmux sessions were alive, port 8200 was free, and every downstream state marker/output path was still clean.

At 00:01 UTC on 2026-08-15, the resumed controller reconciled this persisted state with live processes. Training was healthy at step 113 (loss 0.2558, gradient norm 0.2424, zero NaNs, 28.0k tokens/s) with 92.4 hours remaining. The step-150, continuation, preservation, and finalization watchers were all alive; no candidate/final output collision or premature readiness marker was present. A line-by-line handoff audit confirmed that stock configs leave every custom Pi option unset, scheduled-task denominators come from the saved configs, and the terminal targets remain checkpoint-aligned. The public Terminal-Corpus paper was rechecked directly: its default recipe is 2 epochs, 32,768 tokens, global batch 128, AdamW at 2e-5, cosine with 10% warmup, and it favors unfiltered mixed training. This continues to support the existing same-mixture long run.

Before the step-150 gate, the 32 fixed-seed development images from each suite were prewarmed through the documented sandbox API. Selection reproduced the evaluator's sorted task order and fixed random.Random(0) shuffle; only task.toml image fields and Dockerfile FROM lines were read, never solutions. All 64 containers reached ready in at most 62.8 seconds and all 64 were deleted. An initial API-envelope mismatch (sandbox_id versus the runbook's example id) created 64 pending probes; /resources identified the exact 64 short-ID cohort, and every one was immediately deleted before the corrected run.

The custom empty-end-turn guard was tightened before evaluation to require a nonempty ACP tool-call set, matching its documented purpose: grade a workspace after clean tool activity, but never reinterpret an immediately empty response as completion. The tracked diff archive was regenerated mechanically and its new SHA-256 is 12533f9c06b3fe79f92c193b1a2f90953e44778b0aca3b21e4b8f2166eafc22b.

A second pre-handoff audit fixed a metric-flush race in post_step150_continue.sh: each two-update execution benchmark now requires the step-152 resumable metadata, HF STABLE marker, and logged Step 152 | metric line before terminating its deliberately longer trainer. Without the third predicate a fast poll could kill between the save and metric flush, falsely invalidating every variant. bash -n passed and only the sleeping post-step-150 watcher was restarted; the live trainer was untouched.

1. Finish the first full-parameter mixed SFT gate at step 150.
2. Evaluate step 150 with stock and custom Pi on fixed unbiased samples from both suites.
3. If step 150 validates the recipe, continue the same mixture with a terminal step fixed mechanically before continuation starts. The handoff uses the selected execution variant's measured throughput, a 5% step-time penalty, amortized checkpoint overhead, a 30-minute startup guard, and an 18-hour final-measurement reserve. It caps at step 2,050 (about two epochs). The calculation is independent of candidate rewards and is recorded in state/training-variant-benchmarks.txt and state/continuation-config.txt. Public scaling and curriculum ablations favor more mixed SFT over a component switch.
4. Select the healthy terminal checkpoint a priori (intermediate saves are only rollback points), run the canonical full suites, and package the checkpoint/harness.

At 00:25 UTC on 2026-08-15, before step-150 weights or evaluation outcomes existed, the fixed 1,650/1,800 deadline branch was replaced by the mechanical throughput/time calculation above. This avoids leaving training compute idle if an independently validated GPU-memory variant is materially faster, while the two-epoch cap prevents an open-ended fit. bash -n passed and only the sleeping agentptb-post-step150 watcher was restarted; the live trainer was untouched.

At 00:31 UTC, a resumed-controller handoff audit verified the dynamic terminal path end to end without touching the trainer. Prime-RL's SFT loop always writes both resumable and HF exports at the final max_steps, even when it is not a checkpoint-interval multiple, and the documented CLI syntax accepts the staged --max-steps override. All workspace shell scripts passed bash -n, all Python scripts parsed successfully, and the Prime-RL diff passed git diff --check. The old ac-selective-bench and noactoffload-bench directories contain only dry-validation configs, not stale step-152 artifacts, so the execution gate cannot accidentally reuse a prior benchmark. The manual checklist's obsolete fixed step-1,800 wording was updated to match the already-precommitted dynamic target. Training was healthy at step 125: loss 0.2641, gradient norm 0.2247, zero NaNs, and about 27.6k tokens/s, with roughly one hour remaining to step 150.

At 00:36 UTC, the resumed controller completed another read-only audit of the development and final handoff. Verifiers' eval runner binds each concurrent environment server to tcp://127.0.0.1:0, so the ten-process development gate cannot collide on the serialized config's default port 5000. Its output writer persists one complete episode atomically per scheduled rollout, and the fixed configs have unique output directories. The watcher process-group cleanup, step-152 save/metric predicates, dynamic target propagation, final stock/custom output labels, and scheduled-task denominators remain internally consistent. The live trainer was untouched and reached healthy step 127: loss 0.2558, gradient norm 0.2322, LR 2.30e-6, and 28.1k tokens/s.

At 00:40 UTC, after the supervisor resumed again, all five critical tmux sessions were reconciled against their actual process trees. The clean trainer was healthy at step 128 (loss 0.2551, gradient norm 0.2266, zero NaNs, 28.1k tokens/s, 116.1 GiB/GPU peak), with about 55 minutes projected to step 150. The step-150 evaluation, post-evaluation benchmark/continuation, dynamic checkpoint preservation, and finalization watchers were all alive and waiting on the correct sentinels; port 8200 and every downstream state marker remained clean. A fresh line-by-line audit of the handoff scripts confirmed unique dev output directories, the 120-sandbox aggregate concurrency, process-group-owned inference cleanup, step-152 save-plus-metric gates, and propagation of the mechanically selected terminal step into preservation and packaging.

At 00:47 UTC, the newly resumed controller re-read the persisted state and reconciled it with the live host before taking any action. The corrected trainer was healthy at step 132/150: loss 0.2564, recent-mean loss 0.2600, gradient norm 0.2239, zero NaNs, 28.0k tokens/s recent throughput, and 116.1 GiB/GPU peak memory. All five trainer/evaluation/continuation/preservation/finalization tmux sessions remained alive; port 8200 was free, and no development/final output collision or premature state sentinel existed. A read-only audit of the active handoff reconfirmed process-group-owned inference and benchmark cleanup, the real resumable/HF/metric completion predicates, fail-closed selection files, and dynamic terminal-step propagation. The trainer was not disturbed.

At 00:58 UTC, the active trainer reached step 136/150 with loss 0.2579, recent-mean loss 0.2552, gradient norm 0.2151, zero NaNs, 27.9k tokens/s recent throughput, and 116.1 GiB/GPU peak memory. The resumed controller audited the development launcher, the ten then-staged fixed-sample configs, the precommitted temperature/review selector, the four save-through execution benchmarks, the dynamic continuation target calculation, terminal-checkpoint preservation, finite-tensor packaging gate, and four-way final evaluator. The aggregate development concurrency remains 120 and final concurrency remains 128; stock configs leave every custom Pi option unset. No active process or artifact was changed other than this state record.

At 01:03 UTC, still before step-150 weights or outcomes existed, the development gate gained a conservative scaffold fallback. The SFT-aligned prompt remains the default; true stock Pi can replace it only if stock is non-inferior on each suite, has a positive combined paired advantage, and reaches the same exact sign-test p<=0.10 threshold. Two fixed-sample stock t=0.6 arms were added so the comparison uses identical sampling. They run as a separate 64-sandbox wave after the existing 120-sandbox wave. All six new development/final fallback configs passed the evaluator's dry resolver, shell/Python/TOML checks passed, and the dry output directories were removed. Only the sleeping step-150 watcher was restarted to load the change; the live trainer and other watchers were not touched.

At 01:11 UTC, the newly resumed controller reconciled the full persisted campaign with the live host and left all active processes untouched. The trainer was healthy at step 141/150: loss 0.2536, recent-mean loss 0.2578, gradient norm 0.2115, zero NaNs, 27.8k tokens/s, and 116.1 GiB/GPU peak memory. All five handoff sessions remained live and were waiting on the intended sentinels. A read-only inspection reconfirmed the development selection logic, the four isolated save-through execution benchmarks, deadline-derived terminal step, terminal-checkpoint preservation, full finite-tensor packaging gate, and stock/custom full-suite evaluation. The optional Pi skill path was also inspected; no new skill was added because the concise custom prompt already matches every SFT row and introducing an untrained skill-loading pattern before the paired gate lacked supporting evidence.

At 02:03 UTC, the resumed controller found that step 150 had completed and the trainer had exited normally. The four-way DP candidate server was healthy and the precommitted ten-arm development wave was running at the planned aggregate concurrency of 120; all downstream benchmark, preservation, and finalization watchers remained live. Early traces were explicitly treated as incomplete, not as denominators. They nevertheless confirm the broad motivating failure mode: many step-150 generations still consume the full 4,096-token call budget, and SWE episodes sometimes emit dozens of tool calls in one assistant turn, expanding the observation history until context compaction/termination. At least one early SWE episode had already passed. No selection rule, evaluation config, active process, or checkpoint was changed after observing these partial results.

At 02:38 UTC, the controller again reconciled persisted state with the live host. All ten fixed development evaluators and the four-way step-150 server were healthy, and all four downstream watcher sessions were present. Each evaluation had completed roughly 27--33 rollout attempts, including the first typed retries; the synchronized SandboxError entries occurred at the configured 20-minute episode boundary for the still-running cohort and are not being interpreted as model outcomes. The server continued returning successful requests and all four worker health probes passed. No partial score was used, and no process, config, selection gate, or checkpoint was changed.

Before selection ran, a fail-closed audit found that select_harness.py validated each saved config as 32x1 but did not explicitly require all 32 distinct trace records or identical task keys in each pair. The selector now raises unless both invariants hold. This leaves the precommitted temperature, review, scaffold, and significance rules unchanged while preventing a client crash or mismatched sample from silently influencing selection. Python syntax and the workspace diff check pass; no active evaluator was touched.

The broker /resources endpoint then showed the cause of the synchronized readiness failures: only 33 of 160 shared sandbox pods were running while 127 were pending, despite nominally free aggregate CPU and memory. The candidate server was correspondingly idle and healthy. Before the separate stock-t=0.6 wave launched, its width was reduced from 32 to 16 per arm (64 to 32 aggregate) to avoid repeating this live shared-pool contention. Its exact fixed 32-task samples, sampling parameters, retry policy, and selection rule are unchanged; both TOMLs parse and the watcher passes bash -n.

At 03:03 UTC, the infrastructure-repair path was reconciled with the live selector before either had run. select_harness.py already waits up to eight hours for state/step150-infra-repair-complete, keeping the step-150 server alive and preventing unrepaired sandbox failures from entering selection. The repair backlog was already about 90 episodes, so the repair scheduler was changed from twelve sequential arms to three batches of four arms. Each arm still resumes only its same fixed task indices, preserves model/harness errors, uses an isolated manifest and process group, and retains the six-pass cap. The largest batch requests at most 56 sandboxes, below the documented 128 ceiling. bash -n passed and only the waiting repair tmux session was restarted; no active evaluation, model server, trace, score, or selection rule was changed.

The same observed infrastructure failure was removed from the unattended final path before any final run existed. All sixteen possible final configs now use 8-way TB2 or 16-way SWE concurrency, for 48 aggregate episodes in whichever precommitted four-run combination is selected. run_final_evals.sh then invokes repair_eval_infra.sh, which concurrently resumes only missing rows and typed sandbox/network errors from the same saved runs and fails closed after six passes. Other failures are protected and restored exactly at the parsed-record level; a 32-row semantic prepare/restore round trip passed. All TOMLs parse, the selected aggregate is 48, both shell scripts pass bash -n, and the Python helpers compile. No final task, checkpoint, or score existed when this was fixed.

At 03:36 UTC, the resumed controller found one idle duplicate repair process left from the earlier standalone repair watcher. The active run-2 evaluation watcher already invokes the same repair inline after both fixed waves; allowing both copies to wake on the 12 completed directories could have made them rewrite and resume the same traces.jsonl files concurrently. The duplicate was confirmed to be sleeping in the all-32-rows wait loop with no repair child, then terminated without touching the ten active evaluators or inference server. repair_step150_infra.sh now takes an exclusive workspace flock before editing traces, so any future accidental second invocation serializes safely. The script passes bash -n; all ten evaluators remained alive and the candidate server health endpoint remained 200. A contemporaneous broker snapshot showed 52 running and 105 pending shared sandbox pods, so the already-running 120-wide initial wave is expected to leave typed infrastructure retries for the lower- width repair phase; no score, sample, or selection rule was changed.

At 03:50 UTC, the generic turn-boundary investigation established that the step-150 checkpoint had learned to emit <|im_end|>, while its inherited generation metadata still named only <|endoftext|> as EOS. On the same task-independent tool-call prompt, default serving consumed the full 512-token ceiling and parsed two calls; explicitly stopping on token 248046 consumed 152 tokens and returned one valid call. The pre-selection development attempt was therefore stopped and quarantined as a serving-metadata error without consulting task rewards. scripts/ensure_chat_stop_metadata.py now records both token 248044 and token 248046 as EOS without changing model tensors or tokenizer entries. A fresh server honored that metadata by default (152 tokens, one call), and the normal structured smoke completed in 67 tokens. Terminal preservation and packaging apply and validate the same correction.

At 03:51 UTC, all ten partial pre-fix run-2 directories and their launch logs were moved under evals/quarantined-stop-metadata-run2 and logs/quarantined-stop-metadata-run2; none can enter the selector. A clean step-150 watcher was launched against the identical fixed samples. The twelve arms now run in four sequential three-arm waves at concurrency 8 per arm, so aggregate live sandbox demand is 24. Evaluator signal cleanup was also made idempotent after the deliberate interruption exposed double-wait warnings. The downstream continuation, preservation, and finalization watchers remain fail-closed on the clean gate's completion marker.

At 04:00 UTC, the resumed controller reconstructed the campaign from this file and the live workspace (the supervisor's generic /workspace/state path does not exist in this cell; this workspace's STATE.md and state/ directory are the persisted ground truth). The corrected four-way step-150 server was healthy, its structured smoke used one valid bash call and stopped after 67 completion tokens, and the first clean matched wave was running the three TB2 arms at 8-way concurrency each. The first saved traces were appearing in all three run-2 directories. The continuation, terminal-preservation, and final pipeline watchers were alive and still blocked on the clean evaluation gate; no selection marker or stale infrastructure-repair marker existed. No active process, configuration, task sample, or decision rule was changed during this recovery audit.

At 04:06 UTC, a score-independent audit fixed one missing-row deadlock in the not-yet-run step-150 infrastructure repair. The old precondition waited for all twelve original directories to contain exactly 32 traces before invoking the helper whose purpose includes repairing missing traces. It now waits for the original evaluator processes to exit, requires each saved resume config, creates an empty trace file only if an evaluator failed before doing so, and then lets the existing fixed-index missing/transient repair run. Model, harness, context, and task failures remain protected verbatim. bash -n passes; the active first clean wave was not touched.

At 04:11 UTC, the same not-yet-invoked repair path received the complementary missing-row fix. Its per-arm loop already reported both missing and transient counts, but only entered --resume when the latter was nonzero. A client crash that left no trace row and no recorded exception could therefore reach the strict selector unrepaired. The loop and its terminal assertion now require both counts to reach zero, matching the already-tested canonical-final repair controller. bash -n and the Prime-RL workspace diff check pass; no active evaluator, trace, task sample, or model outcome was changed.

At 04:22 UTC, the resumed controller read the complete persisted campaign and reconciled it with the live processes before intervening. The corrected step-150 server and all four workers were healthy, the structured smoke still showed a clean 67-token tool turn, and every downstream watcher remained blocked on its intended sentinel. The first clean fixed-sample wave had saved 31/32 traces in each of its three TB2 arms; one long episode or typed retry per arm was still active, while the broker had recovered to 37 running pods and only one pending pod. No configuration, trace, task sample, selection rule, model artifact, or active process was changed.

At 04:32 UTC, another recovery audit verified the step-150 checkpoint directly: all four indexed BF16 shards contain the expected 760 tensors (9,409,813,744 elements), the terminal loss/gradient/LR metrics are finite, and the corrected server remains healthy. The three first-wave TB2 arms are still at 31/32; the unmatched rollouts began between 03:54 and 04:06 and remain within the broker runtime's explicit 3,600-second command budget. They were therefore left untouched. The continuation, preservation, packaging, finite-tensor scan, and four-arm full-suite final pipeline were re-read end to end; their sentinels, process-group cleanup, selected-step propagation, fixed denominators, and infrastructure-only retry behavior remain consistent. The deadline is 2026-08-18 20:27 UTC (about 87.9 hours remained at this audit).

At 04:45 UTC, the resumed controller re-read STATE.md, the live state/ directory, the evaluation/runbook references, and the active handoff scripts. The first corrected TB2 wave remained at 31/32 in each matched arm, with all three original evaluator processes still alive and within the declared one-hour command budget. The four-way inference health endpoint reported all workers healthy, the continuation/preservation/finalization sessions remained blocked on the clean selection marker, and no downstream state file had been created prematurely. The active processes and artifacts were left unchanged.

At 05:09 UTC, a fresh controller reconciled the complete in-run record with the live host. The corrected low-temperature TB2 wave had finished all 32 rows in each matched arm with no missing or typed infrastructure failures, and the three matched SWE arms had started normally at eight-way concurrency each. The candidate server and all four workers remained healthy. A reward-blind transport audit found 20 stock Pi HarnessErrors versus 4 aligned-no-review and 6 aligned-review errors; the custom residuals were overwhelmingly deliberately rejected immediately empty turns, while nonempty clean tool-end turns were accepted as designed. Median completion length was 258--261 tokens and only 95 of 1,478 TB2 model calls hit the 4,096-token ceiling, confirming that the portable <|im_end|> stop correction removed the earlier runaway-generation failure. No evaluation outcome, active process, configuration, selection rule, checkpoint, or downstream sentinel was changed.

At 05:20 UTC, the resumed controller re-read the persisted workspace state, evaluation kit, sandbox runbook, active scripts/configs, process tree, and all current artifacts. /workspace/state does not exist in this cell; STATE.md and this workspace's state/ directory remain the ground truth. The low- temperature TB2 arms and stock/no-review SWE arms were complete at 32/32, while the review SWE arm was still running at 28/32; the four-way candidate server and every downstream watcher were healthy and untouched. Completed scheduled- task scores were 3/32 stock TB2, 2/32 aligned-no-review TB2, 2/32 aligned-review TB2, 3/32 stock SWE, and 3/32 aligned-no-review SWE. These are development diagnostics only; no partial result changed the precommitted selection rule or continuation. The stale high-concurrency command in EVAL_CHECKLIST.md was updated to document the already-active four sequential 24-sandbox waves; no executable config or process was changed.

At 05:24 UTC, the controller completed the requested ground-truth recovery from STATE.md, state/, the current run report/checklists, live process tree, GPU ownership, and the evaluation/continuation/finalization scripts. The persisted campaign and host agree: step 150 is healthy and preserved, its corrected chat-stop metadata is active, five of the six low-temperature arms have all 32 fixed traces, and aligned-review SWE has 28/32 with its evaluator still live inside the configured runtime budget. The four-way candidate router reports all workers healthy; the evaluation, continuation, preservation, and finalization watchers are alive, and no selection or continuation marker exists prematurely. No active process, task sample, configuration, selection rule, or artifact was changed during this recovery.

At 05:33 UTC, the newly resumed controller re-read the complete persisted record, the evaluation-kit and sandbox-runbook contracts, the active handoff scripts, current state markers, process tree, GPU ownership, and evaluation artifacts. The host still agrees with the record: the four candidate workers on physical GPUs 4--7 and the router health endpoint are healthy, all four downstream watcher sessions are alive, five low-temperature arms contain 32/32 traces, and the aligned-review SWE arm remains at 28/32 with its original evaluator active inside the one-hour command budget. Neither a selection nor continuation sentinel exists. No active process, evaluation record, model artifact, configuration, task sample, or precommitted decision rule was changed during recovery.

At 05:41 UTC, the final low-temperature arm had advanced to 31/32 traces. Its last retry was waiting on shared sandbox capacity: the broker reported 17 pending versus 3 running pods despite ample aggregate CPU and memory, while the candidate router still reported all four workers healthy. Existing typed retry and fixed-index repair logic covers this infrastructure condition, so no live process or evaluation protocol was changed. The manual evaluation checklist's stale readiness URL was corrected from the DP router's unreliable /v1/models forwarding endpoint to the already-proven /health endpoint plus the subsequent structured-completion smoke; executable automation was already correct and was untouched.

At 06:15 UTC, the resumed controller reconstructed the campaign from the full persisted state, current reports/configs/scripts, live process tree, GPU ownership, and evaluation artifacts. All six corrected low-temperature arms had completed 32/32 fixed traces. The high-temperature TB2 wave was active: stock had completed 32/32, aligned-no-review had saved 25/32, and aligned-review had saved 16/32. The latter two evaluator processes were live and continuing to save traces; the broker had 64 running and one pending pod, and the four-way candidate router reported all workers healthy. The continuation, terminal-preservation, and finalization watchers remained alive and blocked on their intended sentinels. A read-only re-audit confirmed that the continuation launcher does write continuation-launched after atomically recording its chosen config and target. No model, trace, task sample, configuration, precommitted selection rule, or active process was changed.

At 06:21 UTC, a new controller recovered the complete in-run record and audited the evaluation, continuation, preservation, packaging, and final-suite paths. The clean step-150 export, portable dual-EOS metadata, four-way inference server, and all downstream sentinels were consistent and healthy. Aggregate low-temperature results remained development-only: stock/aligned/review solved 3/2/2 of 32 TB2 tasks and 3/3/7 of 32 SWE tasks before infrastructure-only repair. The aligned scaffold sharply reduced unscored harness failures, while the precommitted paired significance gate remains the sole selector. The high-temperature TB2 wave advanced during the audit from 16 to 26 review traces; the evaluator and server were actively using the assigned GPUs. Filesystem headroom was 3.5 TiB, sufficient for the guarded continuation checkpoints. No task-level outcome was used to alter data, training, harness behavior, sampling, or any selection rule, and no active process was changed.

At 06:38 UTC, the controller again left the live high-temperature TB2 wave untouched (stock and aligned-no-review were 32/32; aligned-review was 29/32, with its evaluator alive and the broker reporting no pending pods). A task-identity-blind error audit found that every non-infrastructure custom TB2 HarnessError in the completed low-temperature arms was the same immediately empty first turn (end_turn, no tool statuses); the high-temperature custom arms showed the same residual at lower frequency. It also confirmed that the observed custom context failures occur at the backend's exact 65,537-versus- 65,536 token boundary. Two generic, not-yet-applied hardening candidates are therefore recorded for implementation only after the paired development wave has fully exited: one fresh-session retry for that exact empty-first-turn signature, and a modest advertised context-window safety margin so Pi compacts before the backend boundary. Neither change uses task content, accepts an unchanged workspace as success, or alters stock Pi. No source/config was edited while later development arms could still import it.

At 07:24 UTC, the resumed controller reconstructed the full persisted campaign, read the evaluation and sandbox contracts, and audited the active evaluation, selection, continuation, preservation, packaging, and final-suite scripts. The last high-temperature TB2 retry completed, giving every TB2 arm 32/32 traces, and the three matched high-temperature SWE arms then started at the planned aggregate width of 24. Their evaluators and the four-way candidate server were healthy, while all downstream watchers remained blocked on their intended sentinels. The generic empty-first-turn retry and 512-token advertised-context safety margin remain deliberately unapplied until the complete development and infrastructure-repair gate exits, so no saved arm can mix harness versions. No active process, trace, task sample, selection rule, checkpoint, or executable configuration was changed during this recovery.

At 07:52 UTC, the new controller recovered the complete in-run ledger and live host state. The final high-temperature SWE wave was healthy and advancing, with all nine earlier arms complete and every downstream watcher still blocked on its intended sentinel. A handoff audit found that the generic final-only empty-first-turn retry and 512-token context safety margin had been staged in the submitted patch/configs but the patch was not yet applied to the shared evaluation checkout. A fail-closed agentptb-harness-hardening watcher now waits for the complete development selection and infrastructure-repair marker, waits for the evaluator session to exit, applies and validates only that task-independent opt-in patch, and writes state/harness-hardening-applied. The finalization path now requires that marker, and its sleeping tmux watcher was restarted to load the new gate. Current development and stock configs are unchanged; no live evaluator imported the new behavior.

At 08:14 UTC, the active controller re-read both authorized operational references and recovered the full same-run campaign from STATE.md, state/, the current scripts/configs, and the live process tree. The step-150 server and all four workers remained healthy. All twelve fixed development arms had 32 scheduled records; the precommitted infrastructure-only repair had already cleared the first two batches, including the single custom-review t=0.6 TB2 retry, and was moving to the last batch with seven typed transient records. The continuation, hardening, preservation, and finalization watchers remained blocked on their intended sentinels. No task content, evaluation outcome, active process, model artifact, config, harness source, or precommitted decision rule was changed during recovery.

At 08:19 UTC, infrastructure-only repair finished with all twelve development arms containing the same 32 fixed task keys and no missing or transient episodes. The precommitted selector chose temperature 0.6, aligned no-review. At that setting, scheduled-task scores were 1/32 TB2 and 11/32 SWE for aligned no-review, 3/32 TB2 and 11/32 SWE for aligned review, and 2/32 TB2 and 7/32 SWE for stock. Temperature 0.2 failed its gate (combined official-t0.6-only 9 versus low-t0.2-only 2, exact sign p=0.0654, with a TB2 regression); review failed its selected-temperature significance gate (no-review-only 5 versus review-only 7, p=0.7744); and stock failed the scaffold gate (selected-custom- only 6 versus stock-only 3, p=0.5078, with a TB2 regression). The strong 11/32 SWE signal validates continued broad mixed SFT, while the 32-task TB2 interval remains too wide to support a data-branch change. Full results are in logs/select-harness-step150.json and the twelve saved summaries.

Only after the clean development gate and its evaluator session exited, the task-independent final-only harness hardening watcher applied and validated the opt-in empty-first-turn retry plus 512-token advertised-context safety margin; state/harness-hardening-applied now exists. The candidate inference process released GPUs 4--7, and the guarded post-step-150 handoff began its first isolated save-through execution benchmark at 08:20 UTC. The proven full-AC/CPU-Adam baseline resumed step 150 with the correct dataloader and optimizer state and reached the training loop; no benchmark metric or variant choice existed yet. Continuation, preservation, and finalization remain fail-closed on the benchmark-derived config/target and terminal checkpoint.

At 08:34 UTC, the proven full-AC/CPU-Adam save-through baseline completed both finite updates and wrote valid distributed and HF step-152 checkpoints. Its step-152 loss was 0.2608, gradient norm 0.5164, LR 2.00e-5, peak training memory 116.2 GiB/GPU, and checkpoint-time-adjusted throughput 28.0k tokens/s. This is the valid paired reference for the precommitted 3% execution-variant gate. The micro-batch-10/global-batch-120 benchmark then resumed the same step-150 state and entered its first compiled update on GPUs 4--7; no variant choice or dynamic terminal target exists yet. The continuation, preservation, and finalization controllers remain blocked on those mechanically derived records.

At 08:48 UTC, the resumed controller re-read the complete persisted ledger, both authorized operational references, the live process tree, and the active benchmark/continuation/finalization scripts. The micro-batch-10/global-batch- 120 benchmark had completed finite steps and valid resumable plus HF step-152 saves. Its checkpoint-adjusted throughput was 28,847.85 tokens/s versus 28,002.43 for the paired baseline, a 3.0207% increase that narrowly clears the precommitted eligibility threshold; peak memory was 137.7 GiB/GPU, step-152 loss 0.2639, gradient norm 0.5215, and LR 2.00e-5. The independently resumed micro-batch-12/global-batch-96 benchmark was then loading the same step-150 state on GPUs 4--7. No execution recipe or terminal target is selected until all staged variants finish, and no active process, data, score rule, or model artifact was changed during recovery.

At 09:01 UTC, the micro-batch-12/global-batch-96 benchmark completed two finite updates and valid distributed plus HF step-152 saves. Its checkpoint-adjusted throughput was 26,801.79 tokens/s, below the 28,002.43 baseline, with 159.2 GiB/GPU peak memory, step-152 loss 0.2675, gradient norm 0.5256, and LR 2.00e-5. It is therefore valid but ineligible for the fixed 3% speed gate. The controller released all of its workers and started the independent selective-activation benchmark from the same step-150 state. No execution recipe or terminal target is selected until the remaining staged variants finish.

During the selective benchmark startup, a read-only inherited TORCHINDUCTOR_CACHE_DIR produced nonfatal graph-cache permission warnings. Before any long continuation config was launched, all six possible continuation TOMLs were updated to point TorchInductor at the writable, node-local /tmp/agentptb-torchinductor-continuation directory. Every TOML was parsed afterward and the handoff scripts still pass bash -n. Active benchmark configs and all model/data/optimizer/selection settings were left unchanged.

The selective variant then required a new Inductor graph and the inherited cache error became fatal before its first optimizer update. Before any variant-selection or continuation marker existed, the controller and all of that attempt's workers were stopped; the failed output/log were preserved as outputs/quarantined-cache-permission-ac-selective-bench and logs/quarantined-cache-permission-sft-ac-selective-bench.log. All six execution-only benchmark TOMLs were pointed at the same writable node-local cache and parsed successfully. The identical controller was restarted at 09:08 UTC: it reused the three already-valid save-through arms and immediately relaunched selective AC from the same step-150 state. Its resolved config contains the local override. This is an infrastructure retry only; no task reward, training data, model update, throughput result, or selection threshold was changed.