Predict sparse intent
Egocentric RGB, language, and robot state produce five-point pose sequences and finger targets. Asynchronous action chunks keep execution responsive.
SP-HLM: Sparse-Pose-Driven Motion Generation and Whole-Body Control for Humanoid Loco-Manipulation
1 Shanghai Jiao Tong University2 Moore Threads3 The University of Hong Kong
Five body poses express where to move.
A learned motion prior captures how to move.
THE MOTION INTERFACE
A sparse-pose-driven interface for humanoid loco-manipulation.
Read the abstract +
Humanoid loco-manipulation requires an effective interface between high-level task policies and whole-body control. Existing interfaces typically rely on a single action representation for both task intent and whole-body coordination. This may burden the high-level policy or leave one aspect underspecified. We present SP-HLM, a framework built on a sparse-pose-driven interface. The interface combines five task-critical body poses with a learned latent motion prior to represent spatial intent and whole-body coordination in complementary forms. SP-VLA, a vision-language-action model, first predicts task-relevant sparse poses and finger joint positions from multimodal observations and language instructions using asynchronous action-chunk inference. Given these sparse targets, SP-MGM, a motion generation model trained through two-stage distillation, reconstructs coordinated full-body motion in real time. A pretrained motion encoder maps this motion to a compact latent representation. SP-GMT, a general motion tracker, then integrates the explicit sparse targets with this implicit motion prior for whole-body control. Experiments in simulation and on a real humanoid robot demonstrate improved motion tracking, whole-body coordination, and loco-manipulation success over existing approaches.
METHOD
One interface. Three complementary modules.
Recover coordinated motion
A compressed motion generator uses offline supervised fine-tuning and rollout-aware distillation to follow sparse targets with temporal consistency.
Track intent and motion
A motion encoder supplies a compact latent prior. The tracker combines it with explicit sparse poses to generate body joint commands.
SIMULATED LOCO-MANIPULATION
Whole-body behavior across eight tasks.
52.75%
Overall task success
+19 pp
Over the token-space baseline
8 × 50
Tasks × evaluation trials
TASK 01
Walk forward and pick up the cylinder.
View all simulated task results +
| Method | T1 | T2 | T3 | T4 | T5 | T6 | T7 | T8 | Overall |
|---|---|---|---|---|---|---|---|---|---|
| GR00T · Token space | 22/50 | 7/50 | 24/50 | 20/50 | 8/50 | 9/50 | 19/50 | 26/50 | 33.75% |
| GR00T · Joint space | 11/50 | 11/50 | 20/50 | 19/50 | 8/50 | 21/50 | 13/50 | 23/50 | 31.50% |
| SP-HLM · Direct latent | 20/50 | 6/50 | 20/50 | 6/50 | 16/50 | 6/50 | 6/50 | 13/50 | 23.25% |
| SP-HLM (Ours) | 26/50 | 13/50 | 39/50 | 24/50 | 19/50 | 26/50 | 28/50 | 36/50 | 52.75% |
Both GR00T baselines use the SONIC backend. Direct latent prediction is an SP-HLM ablation.
REAL-WORLD DEPLOYMENT
Real-world humanoid loco-manipulation.
Four tasks on a Unitree G1 equipped with a ZED 2i stereo camera.
Real-world success rates of SP-HLM +
| Task | Successful trials | Success rate |
|---|---|---|
| T1: Object placement | 13/20 | 65% |
| T2: Shelf transfer | 11/20 | 55% |
| T3: Cabinet opening | 5/20 | 25% |
| T4: Trash-bin opening | 8/20 | 40% |
MOTION TRACKING & GENERATION
What makes the sparse-pose interface work?
SP-GMT ablations show that explicit sparse targets and an implicit motion prior provide complementary information for global tracking and local motion fidelity. SP-MGM ablations show that rollout-aware distillation improves sparse-pose adherence, while temporal regularization suppresses high-frequency jitter.
Explicit and implicit references work together
Global MPJPE ↓ · Filtered CLOT-500
Local MPJPE ↓ · Filtered CLOT-500
Rollout-aware distillation improves adherence
End-effector error P95 ↓ · 600 BONES-SEED motions
Generation rate ↑ · Standard inference
SP-GMT comparisons and ablations +
| Method / variant | Progress (%) ↑ | Local MPJPE (m) ↓ | Global MPJPE (m) ↓ | Joint error (rad) ↓ | Global EE error (m) ↓ | Local EE error (m) ↓ |
|---|---|---|---|---|---|---|
| Filtered CLOT-500 | ||||||
| SP-GMT w/ Joint-Space Reference | 99.35 | 0.0305 | 0.0812 | 0.0975 | 0.0879 | 0.0571 |
| SP-GMT w/o Sparse Reference | 99.28 | 0.0390 | 0.4436 | 0.1538 | 0.4506 | 0.0733 |
| SP-GMT w/o Implicit Reference | 98.93 | 0.0659 | 0.1303 | 0.2407 | 0.1470 | 0.1239 |
| SP-GMT w/o Robot Pelvis Pose | 99.33 | 0.0356 | 0.0753 | 0.1330 | 0.0836 | 0.0663 |
| SP-GMT (Ours) | 99.83 | 0.0278 | 0.0676 | 0.0889 | 0.0733 | 0.0516 |
| SONIC | 100.00 | 0.0294 | 0.4177 | 0.1097 | 0.4166 | 0.0509 |
| Humanoid-GPT | 99.93 | 0.0245 | 0.4742 | 0.0572 | 0.4778 | 0.0434 |
| BONES-SEED Diverse-500 | ||||||
| SP-GMT w/ Joint-Space Reference | 90.21 | 0.0315 | 0.0906 | 0.0844 | 0.0908 | 0.0568 |
| SP-GMT w/o Sparse Reference | 92.23 | 0.0418 | 0.5763 | 0.1380 | 0.5804 | 0.0772 |
| SP-GMT w/o Implicit Reference | 91.19 | 0.0645 | 0.1627 | 0.2059 | 0.1678 | 0.1209 |
| SP-GMT w/o Robot Pelvis Pose | 92.15 | 0.0401 | 0.0958 | 0.1309 | 0.1025 | 0.0737 |
| SP-GMT (Ours) | 92.24 | 0.0299 | 0.0797 | 0.0810 | 0.0798 | 0.0553 |
| SONIC | 93.84 | 0.0386 | 0.3181 | 0.1091 | 0.3188 | 0.0609 |
| Humanoid-GPT | 90.10 | 0.0272 | 0.5972 | 0.0573 | 0.5987 | 0.0502 |
SP-MGM ablation results +
| Model | Layers | EE mean (cm) ↓ | EE P95 (cm) ↓ | EE jerk P95 (m/s³) ↓ | Body jerk P95 (m/s³) ↓ | Rate (Hz) ↑ |
|---|---|---|---|---|---|---|
| SP-MGM w/o Stage II | 8 | 8.441 | 12.772 | 1068.626 | 916.724 | 7.634 |
| SP-MGM w/o temporal loss | 8 | 1.915 | 3.420 | 657.125 | 570.246 | 7.322 |
| SP-MGM (Ours) | 8 | 1.812 | 3.212 | 514.337 | 416.945 | 7.813 / 11.494* |
*Text-free inference with TensorRT acceleration.