Humanoid robots need training data that is simultaneously broad, physically precise, and grounded in real object interactions. That combination has been hard to find. A new white paper distributed through Wiley’s IEEE Spectrum content hub introduces HiPHI, a dataset designed to close that gap.
The three-way data problem
Most robot learning pipelines today rely on one of two imperfect sources. Internet video offers enormous behavioral diversity but no access to the underlying joint angles, contact forces, or object states. Laboratory motion capture, by contrast, records movement at very high fidelity, but sessions are typically short and focus on a narrow set of scripted actions. Neither approach handles whole-body human-object interaction well.
The white paper frames this as a three-way trade-off:
- Internet video: diverse but physically imprecise; cannot measure forces, torques, or exact object trajectories.
- Laboratory mocap: precise but narrow; often limited to isolated movements and minimal object contact.
- HiPHI: aims for both breadth and sub-millimeter precision, with synchronized object trajectories and meshes for real manipulation tasks.
What makes HiPHI different
HiPHI is not a small proof-of-concept. It contains 617.5 hours of whole-body human motion captured with optical motion capture at sub-millimeter accuracy. Crucially, 245.7 hours of that total involve human-object interaction, and every one of those interaction segments includes synchronized object trajectories and 3D meshes. That means a robot learning to push, pull, or carry an object can see exactly where the object is and how it moves relative to the person’s body.
To avoid the narrowness that plagues many mocap datasets, the HiPHI team organized coverage using FrameNet, a linguistic framework originally developed to categorize human actions. By mapping motion capture sessions onto FrameNet’s action taxonomy, the dataset systematically spans a broad range of whole-body movements—from locomotion and reaching to complex multi-step manipulation—rather than simply recording whatever volunteers happen to do in a lab.
The white paper also introduces a benchmark suite for measuring two things that matter in humanoid learning: motion diversity and interaction grounding. Motion diversity checks whether the dataset actually covers a wide spread of behaviors. Interaction grounding evaluates how well a model can associate human movement with the precise state of an object. Both metrics move the field away from vague “more data is better” claims and toward measurable dataset quality.
From dataset to real robot
The most practical result in the white paper is the transfer experiment. Researchers trained reinforcement learning policies on HiPHI data and then deployed those policies on a physical Unitree G1 humanoid robot. The paper reports that policy performance improves with the amount of HiPHI training data, suggesting that the dataset’s scale and precision are both contributing to usable skills. Sim-to-real transfer carries the learned behaviors from simulation onto the real robot, where tasks like carrying, pushing, and pulling can be executed with object trajectories that were recorded in the motion capture sessions.
HiPHI was built by Noitom Robotics, which describes its broader project, ModalityNet, as a human-centric data substrate for embodied AI. The white paper is available for free download through the Wiley knowledge hub, though registration is required. For robotics researchers and engineers who have been stuck choosing between noisy internet video and overly narrow lab mocap, HiPHI offers a concrete third option—one that is already showing results on real hardware.