跳转至

Ours residual:可保真复现与可消融的代码架构

目标与边界

ours 不是一个新的 manipulation 算法。它在 Studio 的 PHC-X scene、case、asset、contact label、reset、common rollout、evaluation 和 renderer 接口上,复现已审计 residual 的 policy、observation、action 与 reward 语义。

第一目标是一份 ours baseline config;第二目标才是受控 ablation。一个 ablation 只能改变 明确命名的一项配置字段,并保留父配置、输入/输出维度、配置哈希和全部 feature/reward diagnostics。

Studio 始终拥有运行时接口;参考 task、loader、asset/contact sidecar、renderer 和 evaluator 绝不进入 target runtime。

三层职责

flowchart TB
  I["canonical case/result"] --> M["input.py\nmaterialize once per case"]
  S["Studio PHC-X task\ncase/assets/contact/reset/PhysX"] --> O["ours formulas\npolicy / observation / reward"]
  O --> H["one PHC host class\naction to Studio PD + Studio reward buffer"]
  H --> S
  R["Studio rl-games\nCommonAgent / player / checkpoint"] --> O
  C["baseline config / ablation config"] --> O
  M --> S
  R --> S
  O --> V["independent formula checks + live trace"]

1. Studio runtime layer

submodules/PHC/phc/env/tasks/humanoid_im_studio_residual.py remains the only class that touches PHC task internals. It is responsible only for:

  • obtaining Studio current/reference body state, humanoid DOF state, object link state/qpos/qvel, active joint names/mask, contact labels/regions/forces, timeline and local asset geometry;
  • passing the existing Studio tensors directly to the three formula functions; input.py checks static names, shapes, units and frame counts while loading the canonical case;
  • applying the semantic core's normalized humanoid action through the existing Studio PHC-X pre_physics_step, and writing the returned scalar reward into Studio's reward buffer;
  • accumulating only update-level reward scalars and buffering the small opt-in verification trace.

It must not contain feature concatenation, reward formulas, residual masks, hidden configuration constants, or a second implementation of audited residual math.

2. pipeline/physics/ours/ semantic core

Keep this deliberately small. The old catch-all task.py is removed rather than renamed or exploded into a directory tree. Modules are named after the method's fundamental concepts, not after implementation plumbing:

pipeline/physics/ours/
  input.py          # canonical inputs -> Studio native inputs and case-local config record
  policy.py         # teacher action and residual action composition
  observation.py    # 2596D formula and named slices
  reward.py         # reward formula and named terms
  rlgames.py        # the three framework-required model/agent/player subclasses
  train.py          # inherited-environment PHC command and one phase runner
  run.py            # public `physics.method=ours` entrypoint

policy.py, observation.py and reward.py own the ours method semantics. They are the only files allowed to define their respective formulas. There is no parallel runtime-state schema: the PHC host already owns the live state and calls the formulas with natural tensor groups. input.py owns one-shot canonical materialization, the case-local resolved-config record, and returns the materialized directory after strict validation; it is not imported on the PHC hot path. Small private helpers are preferable to thin dataclasses, custom exception hierarchies or one-file-per-term abstractions.

The semantic core has no import from either reference or vendored reference package. Formula parity is established by test vectors only; all live runtime classes are Studio classes.

3. Configuration and runner layer

  • configs/methods/physics/ours.yaml is the baseline configuration. Controlled ablations are separate, named YAMLs that inherit the baseline and change only their declared fields; there is no profile_id and no loose semantic Hydra knob.
  • The public YAML is the only semantic configuration authority. The run root contains one complete resolved copy and its hash; no second schema, revision protocol or compatibility translation is maintained.
  • train.py passes the resolved values explicitly to PHC Hydra because the vendored learning YAML is only structural host glue. It starts sys.executable and inherits the caller's environment; device visibility belongs to the shell or scheduler.
  • submodules/PHC/phc/run_hydra.py becomes a one-line registration call into pipeline.physics.ours.rlgames. rlgames.py must not import input.py, train.py, run.py or a backend; it contains only the three subclasses required to attach Studio RMS, reward scalars and output finalization to the existing rl-games interfaces.

Baseline config before ablations

The baseline is source-informed but runs wholly in Studio. Source checkpoint provenance and source normalizer state are outside the Studio runtime contract; source residual checkpoints are not loaded.

The baseline config must lock all of the following at once:

Contract Baseline-config rule
Observation named 2596D layout, nonzero Studio-derived bbox when enabled, source feature gates and segment order
Action all source-selected channels, source clip/scale, exact normalized-action/PD ordering
Reward every source term and original additive/multiplicative graph, evaluated from Studio tensors
PPO/player Studio CommonAgent/player/checkpoint interfaces, with all semantically matching current-reference PPO numbers recorded in ours.yaml; no reference custom agent/player path
Timeline explicit teacher/object reference convention with a same-frame trace proof
Articulation Reuse Studio active-joint name/link/DOF/qpos mapping. The baseline 2596D layout requires exactly one active joint; loading a multi-active case fails rather than silently reducing it.
Fixed-object physics Studio fixed-base asset, gravity disabled, joint damping/friction 20/5, and the named Studio 2-mm PhysX contact envelope; every field is persisted in the resolved contract/hash

Studio-owned contact labels and geometry feed the source reward formula through one documented conversion. Studio canonical data supplies boolean hand2 labels [T,2]; reward.py directly materialises a float Studio reference tensor [T,52] by copying the left/right bit only to the five corresponding Thumb3/Index3/Middle3/Ring3/Pinky3 bodies. The other 42 canonical bodies, including both wrists and proximal finger links, remain exactly zero. The enabled hand/other-body and part-specific terms consume that named 52-body tensor, its exact body-name map, live [N,52] body-to-active-region distances and live [N,30] finger-to-active-region distances. The adapter must fail if either the canonical hand2 labels, the documented Studio mapping, or geometry is absent; it must not load a sidecar or zero-fill a missing input.

This preserves formula, threshold, coordinate and action/PD semantics while retaining Studio runtime ownership. A read-only audit of the selected completed b004/b005/b009 jobs proved that the reference converter uses exactly this ten-distal-tip scatter, that its motion [331:383] and VLM sidecar contain the same values, and that the current Studio hand2 files equal the converter's recorded source hand2 files. Those source files remain audit evidence only and are never runtime inputs. Because every selected reference non-hand label is zero, its cg_other reference branch is neutral; the target reproduces that behavior without fabricating geometric labels.

Named slices and reward terms

Every observation block and reward term has a stable identifier, enabled gate, source contract, shape, units and diagnostics. Examples:

obs.body_778
obs.object_links_30
obs.dof_current_153
obs.dof_velocity_153
obs.dof_reference_153
obs.teacher_action_153
obs.active_joint_1
obs.raw_fingertip_force_30
obs.finger_object_pose_450
obs.finger_wrist_pose_450
obs.finger_contact_40
obs.wrist_object_pose_60
obs.object_bbox_144

reward.rb, reward.ro, reward.rcg,
reward.part, reward.part_vel, reward.handle_coarse, reward.handle_normal,
reward.progress, reward.feet_slide, reward.dof_limit, reward.action_rate

For the fullbody layout the mandatory sequence is 778 + 30 + 153 + 153 + 1 + 153 + 153 + 1 + 30 + 450 + 450 + 40 + 60 + 144 = 2596. observation.py rejects a config unless enabled observation blocks sum to exactly 2596. The PHC-X teacher keeps Studio's native local-root SMPL self observation, while the residual policy's 778D block uses the reference-equivalent localRootObs=False encoding. The live adapter computes both views separately from the same Studio body tensors; it never reuses the teacher view as the residual block or changes teacher state/action ownership. reward.py computes the final reward and every permitted named term. rig is an internal neutral factor at the zero-weight baseline and is neither computed nor logged. Studio's existing TensorBoard path records the rollout/update mean of the final reward and every named term; it does not write raw per-step tensors or introduce a second diagnostic pipeline. This makes an ablation a declarative config-YAML change rather than an edit to a monolithic task function.

The phase-progress reward keeps physical initialization and reward supervision distinct. Studio resets the object to the canonical case q0 with zero object DOF velocity. The source-equivalent reward origin is nevertheless q_ref[max(current_phase_start, episode_start)], and the signed opening/closing fraction is clamped to [0,1]. A live reset qpos must not replace that canonical reference origin, even when reconstruction noise makes q_ref[0] differ from case q0.

Verification gates

No baseline or ablation config may enter PPO training before all relevant gates pass:

  1. reference configuration and enabled algorithm contract are recorded;
  2. independent formula checks compute expected observation blocks, composed action, each reward term and final graph without calling the implementation under test a second time;
  3. live Studio trace uses non-synthetic asset/link/contact/timeline tensors and records physics, reference lookup, teacher-input and episode-start frame indices independently;
  4. Studio CommonAgent/player rollout and one PPO update use the selected baseline PPO contract;
  5. a short Studio smoke from fresh Studio initialization completes before any long partial or full training.

Every case-specific checkpoint, smoke, video, closed-loop, train-phase and evaluation record carries the exact input/case/config identity. Formula-only and PHC-X-teacher audits explicitly declare scope=case-independent and are bound to the same resolved config hash; they do not pretend to consume a case. These audit records are verification outputs, not runtime handoffs. run_ours() directly loads one canonical case, materialises the required Studio backend files, validates them, and starts the partial run. All case-specific checks may reuse the same studio_input/ directory. Reference capture is run directly from a separately activated read-only audit environment; target verifiers only read its bounded trace and never rewrite Python/library/device environment variables. Closed-loop parity replays only the captured 153D residual-action rows for at most 64 steps through a normal case-local Studio checkpoint/player. It never copies reference actor weights or RMS into a Studio checkpoint, and the replay path is unavailable unless the bounded trace output is enabled.

pipeline.physics.ours.__init__ must expose public helpers lazily (or remain empty). A dotted import such as pipeline.physics.ours.policy must not eagerly import run.py, a backend, or input.py inside the PHC child process.

All parity captures, config hashes and opt-in per-frame traces live under the run output root, not in source trees. The training hot path performs no per-step JSON or NPZ writes. Long-run gate and resume policy lives in standalone operational scripts, not in the semantic runtime. An ablation report must name its parent config and the one changed contract.

Confirmed design decisions

  • Runtime method, configuration and output identity is ours. Controlled ablations are named config YAMLs, not runtime profiles.
  • Source residual checkpoints and source normalizer state are never loaded by Studio.
  • Baseline human residual action is 153D with residual_start=0, residual_scale=0.5 and residual_clip=1.0. The object receives no residual action; Studio's existing PD execution remains authoritative.
  • The baseline raw 30D fingertip-force observation slot is explicitly zeroed. The separate 40D finger-contact feature remains enabled and is computed from Studio contact/force tensors.
  • The baseline 144D object-bbox feature is enabled and constructed from Studio asset metadata. A case without a valid bbox is rejected before training; it is never silently zero-padded.
  • The baseline reward is the complete reference-v2 multiplicative/additive graph, evaluated only from Studio tensors. It replaces the current Studio parent-reward composition; every weight is an ours.yaml field for controlled YAML ablations.
  • The selected comparison runs were launched on 2026-08-01 with tracked reference HEAD 4edd949a36ad7b4e6e008ed2cbadca52eaf6b1a5. Three surviving pre-launch dirty runtime files are strong mtime/hash-supported candidates for the launch overlay, and their hashes are frozen in output/verification/shared_migration_20260801/selected_reference_runtime_provenance.json. The launcher saved neither a source snapshot nor a Git diff, so these candidates must not be described as a complete or bit-exact reconstruction of the historical dirty overlay. Their reconstructed effects are the articulated-object self-collision filter, optional canonical frame-zero object q0/zero qvel, and 1200-epoch checkpoint cadence; they do not replace the committed reward formula. Later August-4 reference edits are audit history, not the formula oracle for those checkpoints. In particular, those later edits added finger bodies to r_p and changed handle_coarse from the semantic handle region to the whole collision surface; neither edit belongs to this baseline.
  • Baseline ig=0 is strictly neutral: rig is exactly 1, and the runtime does not compute, log, or use an interaction-graph tensor as a reset/termination predicate. A future positive- ig semantic ablation requires an independent named ours YAML and the complete Studio geometry/contact inputs; it must not silently change this baseline lifecycle.
  • Region, geometry and contact metadata are always loaded from Studio. No reference sidecar loader or fallback exists in ours. Every Studio case must expose the metadata required by the enabled reward contract; this includes canonical boolean [T,2] hand labels plus the exact named ten-distal-tip [T,52] conversion, its PHC humanoid body-name mapping, [N,52] body-center-to-region distances and [N,30] finger-body-to-region distances for the part-specific terms. cg_other consumes the live full-body physics contact signal; its selected-reference non-hand labels are all zero, exactly as produced by the audited converter. Any missing required metadata, mapping, or geometry fails normal input loading and is never hidden by changing reward weights.
  • ours reuses Studio's existing asset, active-joint, qpos-reference, contact-label and contact-region loading/mapping methods. It receives only their validated tensors and metadata; the method layer adds observation, policy and reward formulas but has no asset/data loader.
  • The baseline ours config follows the fixed 2596D single-joint contract, like CoDA's one-target reward. Multi-active cases are rejected before training; native multi-joint support is deferred to a separate future extension and is not silently reduced. This means exactly one canonical active joint, not that the Studio object asset may contain only one joint: cases with passive object joints retain their complete Studio qpos/qvel state, reset, physics and recorder columns. The adapter resolves the declared canonical active joint by name to its Studio target DOF; it never selects a first column by position.
  • Teacher state and action come only from Studio's integrated PHC-X tracking interface. ours does not import a reference PHC-X task, PNN runtime or teacher checkpoint; its residual policy uses the selected network/action-composition design and trains from Studio initialization.
  • Studio's canonical timeline is the only timebase for ours observation and reward references. ours does not add a reference-engine t+1 offset; teacher action uses only the timing defined by Studio's tracker. Per-step traces record state, reference and teacher frame indices.
  • The baseline residual policy uses the reference-v2 1024 -> 1024 -> 512 construction, action distribution initialization and residual-composition semantics: separate actor/critic ReLU MLPs, actor-mean constant-zero initialization, fixed log_sigma=-2.9, and no learned sigma. Thus initial deterministic evaluation preserves the Studio PHC-X base action, while training samples use the fixed exploration standard deviation exp(-2.9). All tunable values are ours config fields; architecture or scale ablations require a separate named YAML.
  • The policy input is the fixed reference-v2 2596D segment order documented above. Its implementation is a thin assembly adapter over existing Studio state, reference, local-frame, contact and bbox interfaces; it must reuse those helpers rather than reimplementing asset, geometry, tracking or PHC-X logic. Its two policy object slots are explicitly the live articulated actor root followed by the live active-child rigid body. In particular, the first slot is not substituted with a URDF root rigid body that can carry a fixed local transform; this preserves the reference policy's base-slot coordinate meaning while retaining Studio as the runtime state owner.
  • The PPO baseline copies every current-reference numerical setting whose meaning is identical in Studio CommonAgent; epsilon-greedy and DAgger-KL are not carried over because they are disabled in the selected reference configuration. In particular, the selected reference's effective CLI launch replicates the one Studio canonical case into 512 collection environments, uses episode_length=300 with native early termination, and collects 512 * 32 = 16384 samples per PPO epoch as one 16,384-sample minibatch. The case YAML's numEnvs: 2048 is overridden by the recorded reference launch command and the reference TensorBoard info/epochs step increment confirms 16,384 frames per epoch. These are collection numerics, not mixed-case sampling. Fixed player evaluation remains one environment at Studio frame zero for the canonical 150-frame trace and disables early termination. Training uses the selected run's executed 0.30 m pelvis termination threshold: its generated terminationHeight: 0.15 field is shadowed by a hard-coded 0.30 in the task implementation. In Studio this is measured above the canonical physical ground plane, live_root_z - canonical_ground_height < 0.30; this is identical to the source expression for its zero-ground world and remains correct for canonical cases with a nonzero vertical world origin. PPO ablations use separate named ours YAMLs.
  • The baseline reward weights are exactly the selected fullbody-v2 values: p=30, r=1.5, pv=.25, rv=.02, op=5, or=.1, opv=.1, orv=0, ig=0, cg_hand=5, cg_other=5, cg_all=3, cg_finger=5, eg1=2e-5, eg2=2e-5, eg3=1e-11, part=6, part_vel=.25, progress=2, handle_coarse=4, feet_slide=.25, dof_pos_limits=-1, action_rate=-.005, hand_pos_reward_weight=.25, hand_velocity_reward_weight=.20 and hand_rot_reward_weight=.25. The selected-run r_p averages over exactly 21 key bodies and applies .25 only to its two wrists; it does not append the 30 finger bodies and it divides by 21 rather than renormalizing by the weight sum. Its all-52-body linear-velocity term uses the legacy .20 hand mask, while rotation and angular velocity use .25. Contact, part and handle rewards remain active.
  • The selected reward graph is exactly `rb * ro * rig * rcg * r_part * r_part_vel * r_handle_coarse * r_handle_normal
  • r_progress + r_feet_slide + r_dof_pos_limits + r_action_rate`. Its baseline enables part-specific contact/progress: it replaces only the generic all/finger contact subterms and generic progress, while preserving hand, other-body and contact-energy subterms. All required active-part/handle region, geometry and contact values come from Studio interfaces.
  • r_handle_coarse uses Studio's named active contact-region point cloud, which is the coordinate-converted equivalent of the selected run's semantic handle points. It does not grant coarse credit for approaching an arbitrary point on the door/drawer surface. The separate reference-body proximity weights and object reset use the Studio packaged physics URDF collision surface (512 fixed/root samples plus 1024 active-link samples), transformed through Studio named live/reference link states. None of these terms uses renderer visual geometry. The live contact graph threshold is any(abs(force_component) > 0.1 N); the separate 40D fingertip observation retains its own clipped-force-norm >0.2 N mask.
  • The selected part-contact distance is 0.02 m (handlePartDist in the audited baseline); reward.py applies it to both full-body and finger active-region gates.
  • r_handle_normal is neutral in the baseline because the selected fullbody-v2 config gives it no positive weight. A future named YAML may enable it only when Studio supplies collision geometry, surface normals and the required contact mapping; otherwise loading fails rather than returning a neutral fallback.
  • Normal ours input loading uses Studio interfaces to require exactly one active joint; its resolvable DOF, child-link and qpos/qvel reference; aligned reference frames; active-part/contact-region and finger/body-label mappings; valid bbox metadata; and every required observation/reward shape and unit. Any failure aborts the run: it never reads a reference sidecar, zero-fills, or disables a reward to proceed.
  • Studio's PHC motion export is tracker-local in XY. During the one-time input.py materialisation, ours reads the case-local Studio PHC object's named tracker_local_xy_origin and writes a separate PHC-local articulated reference by subtracting it from only object root/link world positions. This makes object physics, reference links and human-local observation features share the same Studio coordinate basis; orientations, velocities, qpos and all link-local collision/bbox geometry are unchanged. The generic articulated manifest remains the unmodified canonical-world record, and no external transform or source data is consulted.
  • Training reset uses Studio's native reference-state Hybrid initialization, configured to the selected fullbody-v2 semantics (hybrid_init_prob=0.1, hybrid_init_max_fraction=0.5): the probability selects a canonical frame-zero reference reset, while the complementary branch samples a Studio reference state from at most the first half of the motion while still leaving one complete configured rollout before the motion ends. Consequently a 150-frame canonical motion with a 300-step rollout has frame zero as its only legal start. It never selects PHC's generic default/T-pose branch. This changes no runtime ownership: Studio provides reset, canonical timeline and PHC-X state. Deterministic smoke and evaluation reset at Studio reference frame zero. Object physics is reset only by Studio's HumanoidImPassiveObject: the case-defined fixed root pose, the Studio case JSON's explicit initial qpos (q0) for every object joint, and zero root/DOF velocity are installed on every reset. The canonical object trajectory remains noisy pseudo-supervision for observation, reward and timeline diagnostics; ours never writes its qpos/qvel into the simulated reset state. Phase-aware progress keeps this physical reset separate from reward supervision. Its reward origin is the canonical timeline value q_ref[max(current_phase_start, episode_start)]; a live physical q0 must not replace that formula. The humanoid keeps the complete Studio motion state, including root and 153D DOF velocity, for both Hybrid and deterministic Start. This intentionally does not reproduce a reference-loader boundary artifact that hard-codes only its frame-zero DOF velocity to zero while deriving its root/body velocities differently. Studio's PHC motion state is the canonical physical reset contract.
  • Fixed articulated furniture is a cold-start Studio physics contract: the root remains fixed, gravity is disabled, all object joint velocities reset to zero, joint damping/friction are 20/5, and PhysX uses the named ours_contact2mm_sim.yaml 2-mm contact envelope. These are Studio asset/simulator settings, not imported runtime code. They prevent passive joints from moving before contact and preserve the collision envelope used by the 2-cm part-contact reward. The resolved ours config and checkpoint-compatibility hash include all four fields.
  • The three selected Studio cases are independent single-case runs. Each run creates and trains its own policy, optimizer, fresh RMS, checkpoint, TensorBoard stream and evaluation artifacts; there is no mixed-case sampling, shared policy or cross-case resume.
  • ours.yaml is one case-independent method configuration. Studio's existing single-case entry receives exactly one canonical case.json/result.pt pair per invocation; it derives the case-specific inputs and writes an isolated run directory. There are no case-specific method YAML copies or a case manifest inside ours.yaml.
  • Each independent case run evaluates every 100 PPO epochs, retains a full Studio checkpoint every 200 epochs, and also retains Studio's current best checkpoint. The baseline maximum is 1200 PPO epochs; cadence changes require a named ours YAML. The 200-epoch partial is one continuous CommonAgent collector process, and the authorized 200-to-1200 continuation is one further continuous process. Evaluation-boundary snapshots are emitted by that process and evaluated separately; an evaluation must never be implemented by rebuilding the 512-env collector every 100 epochs, because that repeats its seed, Hybrid curriculum and environment state instead of continuing the selected PPO lifecycle.
  • Each case starts with a fresh 200-epoch partial. It automatically continues to the 1200-epoch full run only when canonical input loading, smoke and fixed Studio evaluation/video pass; no non-finite value or anomalous termination occurs; TensorBoard reward terms and total reward are healthy and rising; reward scale and convergence rate are comparable with the selected reference at equal environment frames; and checkpoints, RMS and evaluation artifacts exist. A failed case stops before full training and enters A/B debugging without affecting other cases. Because GPU PhysX continuous net-contact-force readback is not bitwise repeatable, the partial gate additionally requires at least three same-GPU, same-checkpoint Start/frame-zero raw evals. It records median/min/max for reward, MPJPE, active-joint RMSE, opening and contact recall, and saves video only for repeat 1. The final task threshold is diagnostic at epoch 200 because the selected reference itself learns b004 after that boundary; at the final epoch-1200 audit, --require-final-performance requires the median and every repeat to pass with no threshold crossing. The full runner invokes that strict audit on the identity-bound full/train_1200/ours.pth before it can write terminal full_run.json=PASS; a failed median, any failed repeat, or a threshold crossing leaves the completed training artifacts but records terminal verification FAIL. scripts/physics/run_ours_training.sh is the single post-pretraining operational command: it runs the partial, the repeated fixed-eval audit, the fail-closed partial gate, and the full continuation only when that verifier exits PASS. Its final argument is the integrity-bound case-to-reference baseline manifest. The semantic runtime does not import or fork verification scripts.
  • The source-relative terminal audit above is an integration/replication diagnostic, not the project-level task-success definition. The current state machine in docs/experiments/physics_metrics.md is a DEV implementation snapshot, not a paper-frozen contract; articulation outcome, contact/full-horizon wrapper and thresholds remain pending human selection. Current proximity plus marginal-force telemetry must not be called verified same-link pair contact. Historical source rollouts without the chosen formal evaluation may only be used as explicitly labelled opening proxies when comparing absolute displacement and normalized-progress candidates.
  • Full20's lifecycle field task_status is the terminal opening gate, not the paper-level complete state-machine Binary Success. It uses the same target-owned signed-opening helper as offline Studio/source analysis: the origin is canonical q_ref[0], the sign follows the maximum active-phase reference excursion, and opposite-direction displacement cannot count as opening. Evaluation validates but never rewrites the training-stage partial-gate record or its hash.
  • The frozen teacher is inherited only from Studio's existing physics.phcx configuration and integrated PHC-X tracker. ours.yaml has no duplicate teacher path. The residual policy starts fresh and has no residual-checkpoint, source-config-path or source-normalizer input.
  • Observation normalization has one ours.yaml boolean, normalize_input. The baseline enables Studio's fresh running statistics; normal checkpoint resume restores only that Studio checkpoint's own state. The method has no external-RMS mode and does not add a custom snapshot or update cadence. It inherits the current Studio CommonAgent/rl-games model lifecycle: rollout is evaluated with the current case-local RMS, then the Studio RMS is updated by each PPO training forward. With the baseline's one 16,384-frame minibatch and six mini-epochs, a fresh one-update checkpoint therefore has RMS count 1 + 6 * 16,384 = 98,305. This matches the selected clean reference's six-update cadence while retaining Studio's own RMS implementation and checkpoint restore path.