跳转至

D3D-HOI V41R HOI Align Tuning Guide

Last updated: 2026-07-10 Asia/Shanghai.

This guide is the operational protocol for V41R-based HOI-align tuning. Keep historical run status in docs/results/reconstruction.md; keep this file as the stable tuning habit and decision checklist.

Scope

  • Source: V41R full20 per-case recon/output/result.pt.
  • Initial target set: failure5 before full20 expansion.
  • Goal: improve hand-object contact without silently buying it through object drift, bad q, bad human 2D alignment, hand-pose distortion, or jerk.
  • Default contact labels: mano_binary raw labels. Do not apply source-distance contact gates unless the run is explicitly named as a gate diagnostic.

Hard Rules

  1. Start every HOI-align tuning run from the V41R source result.pt.
  2. Set reconstruction.optimization.resume_global_opt_result_path to the per-case V41R result path.
  3. Symlinking or copying recon/preprocess is not enough.

  4. Do not chain from an older HOI-align output unless the run is explicitly a diagnostic/rejected ablation.

  5. Verify source preservation after every run.

  6. Compare source vs output object translation, rotation, scale, and q trajectory.
  7. q must not collapse to zero from MLP initialization.

  8. If object/q/human are supposed to be fixed, verify the delta is numerically zero or explained.

  9. Use only the two standard qualitative figures for tuning review.

  10. Do not produce HOI/keyframe MP4s by default.
  11. Do not use montage-style figures or ad-hoc alternate visualizations as tuning evidence.

  12. The only default visual outputs per case/run are:

    1. Keyframe 3D multiview sheet from scripts/experiments/visualize_contact_keyframes_1x2.py.
    2. A 4-column renderer-localfit contact-context sheet when explicitly needed.

  13. Metrics alone are insufficient.

  14. A candidate only matters if metrics and the two static figures agree.
  15. A low contact number is not accepted if q starts opening too early, object mask/IoU worsens, hand pose deforms, or human 2D/jerk drifts.

Default Diagnostic Figures

Figure 1: Keyframe 3D Multiview Sheet

Use this for global hand/object alignment over selected contact keyframes. It is a 2-row x N-column sheet; each keyframe cell contains incam and right+45 views.

Template:

CUDA_VISIBLE_DEVICES=0 /DATA/intern/hoi4d/miniconda3/envs/arthoi4d/bin/python \
  scripts/experiments/visualize_contact_keyframes_1x2.py \
  --preprocess-dir <case_recon_preprocess> \
  --contact-ref <contact_ref_mano_binary_palmar_k8.npz> \
  --result <case_recon_output_result.pt> \
  --object-source-type partnet \
  --object-source-uri data/partnet/<partnet_id> \
  --output-dir <run_visualizations>/contact_keyframes_1x2 \
  --human-source result \
  --max-vis-frames 5 \
  --name <run_case_name>_keyframes \
  --sheet-name <run_case_name>_keyframe_3d_multiview_sheet \
  --sheet-rows 2 \
  --sheet-cell-width 1080 \
  --sheet-pad 18 \
  --device cuda

Expected primary output:

<run_visualizations>/contact_keyframes_1x2/<run_case_name>_keyframe_3d_multiview_sheet.png

Do not pass --write-video unless the user explicitly asks for video.

Figure 2: 4-Column Contact-Context Sheet

Use this for local contact geometry. Each row is one keyframe with:

RGB crop | incam local 3D | BEV local 3D | side local 3D

Color semantics:

  • Yellow: active hand vertices.
  • Blue: nearest object contact-region matches and links.
  • Red: local object contact-region samples.

The historical renderer-localfit utility was removed with the old tuning suite.

Numeric Gates

Treat these as current working gates for failure5 diagnostics, not final paper thresholds.

  • Contact distance: target mean <1-2cm; p90 should not expose large misses.
  • Penetration: should stay zero or negligible.
  • Object visual alignment:
  • IoU/keyframe mask should not visibly worsen.
  • Object global translation/rotation/scale deltas must be justified by the experiment.
  • q/articulation:
  • q limits must stay valid.
  • For b004-0017, first closed frames are a hard diagnostic: first-10 q should not open early.
  • Middle/peak q should not be sacrificed if the source under-opens relative to GT.
  • Human:
  • Hand pose must not become visually distorted.
  • Hand 2D drift should stay guarded; current working p90 gate is around <30px.
  • Human jerk should remain close to V41R unless the run is explicitly a rejected upper-bound diagnostic.

Standard Tuning Loop

  1. State the hypothesis in the run name.

  2. Examples: qfixed_object6d_scale, qanchor_sweep, contact_weight_sweep, early_q_guard.

  3. Run the smallest meaningful set first.

  4. For object/q mechanism debugging, start with b004-0017_cad7221.

  5. For acceptance, evaluate failure5 before considering full20.

  6. Evaluate raw contact labels.

  7. Use all active mano_binary contact frames.

  8. Do not use 3cm/15cm source-distance gates as the default objective or eval.

  9. Compute preservation diagnostics.

  10. Source vs output object translation/rotation/scale.
  11. Source vs output q trajectory and sparse GT q error when applicable.

  12. Human hand-pose deltas, 2D drift, and jerk if human is optimized.

  13. Use the unified metric protocol.

  14. Full object/contact metrics: scripts/experiments/evaluate_d3dhoi_full270_recon.py.
  15. q/first-10/sparse-GT metrics: use the same q-summary logic as the R42/R43 scripts, not a new ad-hoc metric.
  16. Source preservation: compare source and output transforms directly in the evaluation result.

  17. Keep contact_ref, selected frames, and IoU/contact summary fields identical to earlier comparable runs.

  18. Generate exactly the two static figures above.

  19. First figure: 2-row keyframe 3D multiview sheet.
  20. Second figure: 4-column renderer-localfit contact-context sheet.

  21. No MP4 unless explicitly requested.

  22. Decide with gates.

  23. Accept only if contact, q, object visual alignment, hand pose, and motion plausibility all pass.

  24. If a run improves contact but fails one gate, record it as a diagnostic, not a candidate.

  25. Update the active state file.

  26. Record run root, key metrics, visual paths, conclusion, and next experiment.

Current Lessons From R41-R51

  • R41 object translation + scale + q, human fixed, no contact gate:
  • Contact improves strongly, but object alignment and early q fail.

  • Read: object/q can explain the contact, but not safely.

  • R42 q-anchor sweep:

  • Increasing scalar q anchor did not fix early opening.

  • Read: the problem is not only insufficient scalar q regularization.

  • R43 contact-region weight sweep:

  • Lowering contact weight improves IoU and can keep contact mean below 2cm, but still does not recover closed first frames.
  • contact_region_topk=50000 is the current conservative scalar diagnostic default.

  • Read: scalar global contact weight cannot simultaneously keep the start closed and retain useful middle/peak opening.

  • R44 q fixed, object 6D + scale:

  • q fixed exactly.
  • Contact improves from V41R but stays above the target: 3.578/6.089cm mean/p90.
  • Object compensation is nontrivial: translation 3.78cm, rotation 5.15deg, max scale delta 0.0132.

  • Read: 6D pose + scale alone is insufficient for grasp contact below 2cm; controlled q/articulation correction is needed.

  • R45 mask-guard sweep from R43 contact50000:

  • Only loss.mask.value changed: 160/400/800 vs R43 80.
  • IoU improves clearly as mask guard increases.
  • Contact mean stays below 2cm.
  • First-10 q does not improve and slightly worsens.

  • Read: stronger silhouette/mask guard is useful for object visual alignment, but it is not the main solution for early q opening.

  • R46 active-part track3d diagnostic:

  • The exact V41R track term did not materially change first-10 q, contact, IoU, middle q, or peak q.
  • The temporary HOI-align track path was withdrawn at user request.

  • Read: do not use track loss in subsequent HOI-align tuning.

  • R47 absolute-q acceleration-smoothness sweep:

  • Only loss.smooth.joint changed: 600/1800/6000/9000 vs R43 60; all track losses stayed disabled.
  • The loss acts on q_abs=q_base+delta_q, not on delta_q alone.
  • Increasing the weight lowers measured q acceleration, but first-10 q remains around 6.1-6.2deg, while middle/peak q and contact remain essentially unchanged.
  • smooth=9000 slightly worsens IoU.

  • Read: acceleration smoothness cannot remove a constant or nearly linear initial offset and has no frame-0/1 boundary term. Keep smooth=60; do not continue increasing this scalar.

  • R48 direct-q anchor x smoothness grid:

  • q is initialized from the V41R absolute trajectory and optimized as independent per-frame direct parameters; no residual MLP is created.
  • The existing q anchor remains well-defined as ||q_direct-q_V41R||^2, and q smooth remains the absolute-q acceleration term.
  • Direct q fixes first-10 drift across anchor={60,240,960,3840} x smooth={60,9000}, but contact remains around 3cm and middle opening is under-corrected.
  • Anchor and smooth changes have negligible influence compared with the parameterization switch.

  • Read: direct q is preferable for temporal locality, but its current 0.001 articulation LR / 200-step budget is insufficient. Test direct optimization capacity next; do not treat any R48 grid cell as accepted.

  • R49 direct-q articulation-LR sweep:

  • Only object_articulation_lr changed: 0.001/0.0015/0.002/0.003/0.005/0.008/0.012/0.02; direct q, anchor60, smooth60, contact50000, mask80, 200 steps, freedoms, and disabled track losses stayed fixed.
  • LR controls direct-q capacity. Raising it increases middle/peak q and improves contact, but also increases first-10 opening.
  • LR 0.002 gives the best IoU (0.830/0.782) and first10 2.396deg, but contact remains 2.223cm.
  • LR 0.003 is the first contact-passing point (1.708cm) and keeps good IoU (0.820/0.770), but first10 rises to 4.536deg.
  • LR >=0.005 reaches R43-like middle/contact but loses the direct-q early benefit; LR 0.02 also becomes visibly jagged in q acceleration.

  • Read: no scalar articulation LR passes both strict early-closed and <2cm mean-contact gates. Keep direct q, but do not accept R49 or expect a narrow LR-only sweep to remove the measured trade-off.

  • R50 human-only continuation from R49 LR 0.002:

  • Run one 200-step human-only phase with object rotation/translation/scale/q locked, contact50000 unchanged, and the established R39A2 human hand-pose, 2D, and temporal guards.
  • Contact improves from 2.223/5.541cm to 1.742/4.327cm mean/p90 while IoU, q, object pose, and scale remain unchanged.
  • Human-joint 2D drift is 4.66px p90; active-hand pose delta is 2.94deg p90; hand jerk remains within 1.002x of source.

  • Read: use the conservative direct-q LR 0.002 object result, then let a guarded human-only phase absorb the remaining few millimeters. Do not force object-only contact below 2cm by increasing q LR to 0.003 on this case.

  • R51 failure5 200+200+200 chain with fixed residual translation:

  • Phase 2 reaches <2cm contact only on b001; phase 3 worsens contact in 3/5 cases and generally loses IoU.
  • R51 did not reproduce R49 on b004 because R49's object stage had a trainable shared residual translation despite optimize.translation=false.
  • Read: optimize.translation=false does not imply a fixed object when object_residual.translation_mode=shared_xyz. Treat residual translation as an explicit optimization freedom and report its source delta. Reject R51 and first reproduce the R49 shared-translation baseline before tuning phase count or duration.

Current Direction

The next useful design should keep direct q and explicitly address the per-frame objective conflict shown by R49:

  • keep early closed frames near V41R/GT closed q,
  • allow q correction where there is object-motion/contact evidence,
  • keep object global pose and scale guarded unless the run is explicitly object-diagnostic,
  • preserve R38/R39-level human keypoint2D and hand-pose guards,
  • validate every iteration with the two standard figures and numeric gates.

R45 rules out "just increase mask loss", R47 rules out "just increase absolute-q acceleration smoothness", R48 shows q anchor/smooth are not the controlling direct-q variables over the tested ranges, and R49 shows a single global direct-q LR only moves along the early-q/contact trade-off. R50 shows the remaining contact gap can instead be handled by a guarded human-only phase. Do not return to residual MLP or track loss.