HDMI is a novel framework that enables humanoid robots to acquire diverse whole-body interaction skills directly from monocular RGB videos of human demonstrations.
This repository contains the official training code of HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos.
- Release hdmi training code
- hoi motion datasets
- Release pretrained models
- Release sim2real code
# setup conda environment
conda create -n hdmi python=3.11 -y
conda activate hdmi
# install isaacsim
pip install "isaacsim[all,extscache]==5.0.0" --extra-index-url https://pypi.nvidia.com
isaacsim # test isaacsim
# install isaaclab
cd ..
git clone [email protected]:isaac-sim/IsaacLab.git
cd IsaacLab
git checkout v2.2.0
./isaaclab.sh -i none
# install hdmi
cd ..
git clone https://github.com/EGalahad/hdmi
cd hdmi
pip install -e .
Teacher policy
# train policy
python scripts/train.py algo=ppo_roa_train task=G1/hdmi/move_suitcase
# evaluate policy
python scripts/play.py algo=ppo_roa_train task=G1/hdmi/move_suitcase checkpoint_path=run:<wandb-run-path>Student policy
# train policy
python scripts/train.py algo=ppo_roa_finetune task=G1/hdmi/move_suitcase checkpoint_path=run:<teacher_wandb-run-path>
# evaluate policy
python scripts/play.py algo=ppo_roa_finetune task=G1/hdmi/move_suitcase checkpoint_path=run:<student_wandb-run-path>Please see github.com/EGalahad/sim2real for details.
