跳转至

Generated Articulated Asset Video Tracking Plan

Date: 2026-06-19

This is a new route proposal. It does not replace the historical D3D-HOI PartNet retrieval route summarized in docs/results/reconstruction.md.

1. Motivation

The current D3D-HOI tuning route uses a retrieved PartNet asset and optimizes rigid pose, scale, and articulation against video evidence. V27D/V36A show a core limitation of this setup: even with correct active-part masks, the rendered active link can disagree with the observed object because the CAD asset shape, size, handle geometry, and local detail do not match the real instance. In that regime, mask/depth/track/contact losses can become ambiguous. Lowering false-open can easily become under-opening, and tracking losses can bind to the wrong part.

The proposed pivot is to make the object asset more instance-matched before video tracking starts:

  • Use image-to-3D or related asset generation to produce a static textured mesh from the observed object.
  • Use an articulation method such as Instruct-Particulate to convert that mesh into an articulated, simulator-importable asset.
  • Keep the asset geometry and texture fixed during video reconstruction.
  • Use the real video only to estimate/refine object rigid pose, scale, articulation trajectory, human-object alignment, contact, and later physics consistency.

The goal is not to claim a new image-to-3D generator. The paper contribution would be video-grounded articulated HOI tracking and interaction reconstruction given generated articulated assets.

2. Relation To Existing Docs

Closest existing planning docs:

  • docs/experiments/index.md: broad paper experiment plan for D3D-HOI reconstruction, HOI alignment, and physics refinement.
  • docs/research/baselines_and_evaluation.md: earlier Q&A about ArtHOI, baselines, metrics, and dataset comparability.
  • docs/results/reconstruction.md: historical D3D-HOI PartNet retrieval evidence. It explicitly assumes PartNet assets only and PartNet-format active joints.

This file is intentionally separate because the new route changes the object asset assumption. It should be evaluated as a new branch of the paper story, not as another V27D articulation-loss variant.

3. Core Task Definition

Input:

  • Real monocular HOI video.
  • Object crop or keyframe image(s).
  • Human motion/masks/depth/tracks from the existing preprocessing stack.
  • A generated articulated object asset: visual mesh, part segmentation, joint graph, joint axes, limits, and simulator import files.

Output:

  • Object rigid pose trajectory.
  • Object scale.
  • Articulation q trajectory.
  • Human-object alignment and contact timing/locations.
  • Physics-ready replay or refinement state.

Out of scope for the video optimizer:

  • Geometry refinement.
  • Texture refinement.
  • Learning a new image-to-3D model.
  • Treating generated asset errors as free variables to silently absorb tracking failures.

RGB losses can be used, but only as pose/q/contact grounding terms for a fixed asset. They should not become implicit mesh/texture optimization.

4. Candidate Asset Frontends

4.1 Hunyuan3D / Similar Image-To-3D Models

Role: produce a textured static mesh from an object crop/keyframe.

Why it is useful:

  • Directly attacks the PartNet mismatch problem.
  • Gives texture, not just geometry, so robust RGB or feature losses become more meaningful.
  • Current Instruct-Particulate demo code was designed around this kind of off-the-shelf image-to-3D mesh frontend.

Limitations:

  • The mesh is usually monolithic and not articulated.
  • Generated back-side geometry and occluded handles can be hallucinated.
  • Canonical orientation, scale, and physical collision quality still need validation.

4.2 SAM 3D Objects

Role: possible alternative static asset frontend, not an articulation solver.

Meta's SAM 3D Objects reconstructs object shape, texture, and layout from a single masked image and is designed for natural images with clutter/occlusion. That makes it relevant to real D3D-HOI crops. However, for this pipeline it is only a drop-in replacement for Hunyuan3D if it exports a clean mesh format that Instruct-Particulate or our asset adapter can consume.

Expected difference versus Hunyuan3D:

  • It may be better for in-the-wild masked objects, occlusion, and scene-context recovery.
  • It does not by itself solve part segmentation, joint axis estimation, URDF export, or HOI video tracking.
  • It can also be used as a mask/layout/crop helper even if Hunyuan3D remains the mesh generator.

The practical recommendation is to treat Hunyuan3D and SAM 3D as swappable static-mesh candidates and evaluate them by downstream q-surface/video-tracking behavior, not by mesh screenshots alone.

4.3 Instruct-Particulate

Role: convert a static mesh plus a kinematic specification into an articulated asset.

Its input is not only an image. It expects:

  • A static 3D mesh.
  • A target kinematic specification: part names/descriptions, connectivity, joint types, and optional point prompts.

Its output is closer to what this project needs:

  • Part segmentation.
  • Joint motion parameters.
  • Animated GLB / visualized axes.
  • URDF-style export payload with per-part meshes and joints.

Important caveat: the exported asset is simulator-importable, but not necessarily final high-quality physics content. Collision meshes, mass, inertia, joint limits, and scale may still need a deterministic adapter/validator before SAPIEN or another simulator can trust it.

4.4 Articraft And Particulate

Articraft is strong for scalable articulated asset generation through an agentic program-writing system. It is best viewed as a way to build or expand a simulation-ready articulated asset library, or as a source of category priors and validation ideas. It is not the most direct solution for reconstructing a specific real video instance from an object crop.

Particulate predicts articulated structure from a static 3D mesh without the same explicit test-time kinematic control. It is useful as a comparison or fallback, but Instruct-Particulate is better aligned with our need to specify "this microwave has body + door with a revolute joint" and then validate the result on video.

5. Kinematic Specification Policy

For a first D3D-HOI gate, manual specs are acceptable. Example for a microwave or cabinet-style case:

{
  "links": [
    {"id": 0, "name": "body"},
    {"id": 1, "name": "door"}
  ],
  "joints": [
    {"parent": 0, "child": 1, "type": "revolute"}
  ]
}

For arbitrary videos, hand-writing this per case should not be the final pipeline. The scalable route should be:

  1. Generate or reconstruct a static mesh from the object crop.
  2. Render several canonical views of that mesh.
  3. Ask a VLM to propose the kinematic graph: links, names, parent/child connectivity, joint types, and optional point prompts.
  4. Run Instruct-Particulate with that specification.
  5. Validate automatically: single connected tree, sane part sizes, axis inside or near the object, joint-limit plausibility, no tiny disconnected active part, and q-sweep render evidence on keyframes.
  6. Repair or simplify when validation fails, e.g. fall back to one active revolute/prismatic link for common categories.

"VLM" here means a vision-language model/API call, not a virtual machine. We do not need to open a VM just to write the kinematic spec. For early experiments, manual JSON bypasses the VLM step. For scale-up, we can use an API or a local VLM depending on reproducibility and policy constraints.

6. Proposed Pipeline

Stage A: Asset Proposal

  • Select a keyframe where the object is visible and minimally occluded.
  • Segment/crop the object with existing masks, SAM/SAM3D, or manual diagnostic masks if needed.
  • Generate candidate static meshes with Hunyuan3D, SAM 3D Objects, or another image-to-3D frontend.
  • Normalize orientation and remove obvious floating fragments.

Deliverables:

  • asset_raw.glb
  • asset_clean.glb
  • object crop/mask used for generation
  • asset quality sheet with front/side/top renders

Stage B: Articulation

  • Provide a kinematic spec manually for the first gate.
  • Run Instruct-Particulate.
  • Export part meshes, joint axes, limits, and URDF-like structure.
  • Build a deterministic adapter into the project renderer/simulator.

Deliverables:

  • kinematic.json
  • seg.npy
  • mesh_parts_with_axes.glb
  • per-part visual/collision meshes
  • simulator import config

Stage C: Video Grounding

Optimize only tracking variables:

  • global scale
  • object rigid pose trajectory
  • q trajectory
  • human-object temporal alignment if needed
  • contact variables / contact windows

Candidate losses:

  • silhouette and part-mask loss
  • robust depth loss
  • RGB or feature loss inside visible object regions
  • rendered-flow or keypoint/track consistency when binding is validated
  • contact/grasp alignment
  • temporal smoothness and physics-consistency losses

Key rule: do not silently modify mesh geometry or texture to make the video loss happy.

Stage D: Physics Readiness

  • Separate visual mesh from collision proxy.
  • Generate convex or simplified collision proxies when needed.
  • Assign explicit mass/inertia defaults and record them.
  • Validate joint axes/limits in SAPIEN.
  • Run replay/rollout checks before physics refinement.

7. First Minimal Gate

Recommended first target: a V27D/V36A failure where PartNet mismatch is obvious and the open state is hard for the current renderer, e.g. b007-0017.

Gate 1: asset q-surface diagnostic.

  • Keep camera/rigid pose fixed as in V27D for a diagnostic.
  • Sweep q on selected closed/open/tail keyframes.
  • Compare rendered mask/depth/RGB-feature loss curves for PartNet versus the generated articulated asset.

Acceptance signal:

  • Closed frames prefer q near closed.
  • True open frames prefer a much larger q than the PartNet/V36A diagnostic did.
  • The loss surface is not flat or dominated by wrong static geometry.
  • Visual overlays show active part alignment, not just lower false-open.

Gate 2: single-case tracking.

  • Optimize pose/q/scale with the generated articulated asset fixed.
  • Compare against V27D and V36A on motion, false-open, false-close, predicted range, and jerk.

Acceptance signal:

  • Lower false-open without high false-close.
  • Motion improves or at least does not regress.
  • Predicted range remains plausible.
  • No large jerk/contact regression.

Gate 3: small category gate.

  • Repeat on 3-5 cases with the same category or mechanism.
  • Only then consider a larger protocol.

8. Paper Story If This Works

Old route:

  • Retrieval-based articulated CAD reconstruction is brittle when the retrieved CAD does not match the real instance.
  • Stronger masks/tracks can expose the mismatch but cannot always solve it.

New route:

  • Generate or reconstruct an instance-matched static object asset from the video/keyframe.
  • Convert it into an articulated simulator-ready asset with a controllable kinematic graph.
  • Track pose, q, human alignment, and contact from video while keeping the generated asset fixed.
  • Use physics only after the video-grounded state is geometrically plausible.

Possible contributions:

  • A generated-asset-to-HOI tracking pipeline for articulated objects.
  • A deterministic adapter/validator from generated articulated meshes to the renderer/simulator.
  • Video-grounded pose/q/contact optimization that uses RGB/depth/mask/features without changing geometry.
  • Automatic or semi-automatic kinematic-spec proposal and validation.
  • A diagnostic protocol showing when retrieval CAD, part masks, and generated assets do or do not solve articulation ambiguity.

Baselines:

  • V27D PartNet retrieval baseline.
  • V36-style oracle/GT part-mask diagnostic.
  • Generated static asset without articulation-aware tracking.
  • Instruct-Particulate asset with naive q/pose initialization.
  • Ours with/without RGB-feature grounding, contact, and physics validation.

9. Risks

  • Generated meshes may look plausible but have wrong dimensions, missing handles, sealed doors, or bad back-side geometry.
  • Texture mismatch can make raw RGB loss harmful. Prefer robust masked photometric or feature loss and always gate with geometry diagnostics.
  • Instruct-Particulate can segment the wrong active part or estimate an axis that is visually plausible but physically wrong.
  • URDF export is not automatically high-quality simulation content. Collision, inertia, mass, and limits need explicit handling.
  • VLM-generated kinematic specs can be wrong. The system needs validators and repair paths, not blind trust.
  • Reviewers may see the route as a system pipeline unless the tracking, validation, and interaction reconstruction contributions are clearly isolated and ablated.

10. Immediate Next Steps

  1. Pick one high-value failure case such as b007-0017.
  2. Produce one or more static mesh candidates from the same object crop: Hunyuan3D first, optionally SAM 3D Objects as a comparison.
  3. Manually write the minimal body-door revolute spec for this first gate.
  4. Run Instruct-Particulate and export the articulated asset.
  5. Build the smallest renderer/simulator adapter needed for q-sweep overlays.
  6. Run the q-surface diagnostic before any full optimizer work.
  7. If the generated asset still cannot make open frames prefer open q, reject this case as not solved by asset generation and inspect whether the failure is generator quality, articulation segmentation, axis, camera/pose, or occlusion/contact evidence.