Very poor training performance with GR00T on the PhysicalAI-Robotics-GR00T-Teleop-Sim dataset for robocasa-gr1-tabletop-tasks

[4133223]

We are using GR00T for training on the dataset mentioned in robocasa-gr1-tabletop-tasks, which is available at https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-Teleop-Sim. The training parameters are as follows:

We have tried loading either a single dataset or multiple datasets for training, but the model performance is very poor. When loading multiple datasets for training, the test success rate of most tasks is 0, and the maximum for a small number of tasks is only around 30%.

[4133223]

The test output videos are as follows:

ccd33b4f-ac7f-4880-a127-109d7560d4a1_success0.mp4

be3cf1a9-c114-468f-8299-54a0f39f80d9_success0.mp4

[34328]

I meet the same problems, i just SFT in ''gr1_unified.PnPCanToDrawerClose'' task dataset，The success rate is also very low，10%？

[4133223]

Perhaps it's a problem with the model. They've recently updated it, and I need to give it a try immediately.

[4133223]

The pre-trained model remains unable to accomplish the tasks within the robocasa-gr1-tabletop-tasks benchmark. I contend that fine-tuning is imperative.

[yeshas1994]

I have also tested finetuning PnPCanToDrawer (all 1000 trajectories) for 30,000 - 60,000 steps and consistently get around a 35-45% success rate.

[4133223]

@yeshas1994 My conclusion aligns with yours. I also fine-tuned the latest model specifically for the single task of PnPCanToDrawer, and the success rate hovers around 40%. Of course, RoboCasa's success criteria can sometimes be inconsistent—it might misclassify a successfully completed task as a failure. When I fine-tuned the model on multiple tasks simultaneously, I noticed a clear trend: the more tasks I included, the worse the performance on individual tasks became. I'm curious—were the results in the paper achieved by fine-tuning on all tasks concurrently?

[youliangtan]

Similar issue: #249

With my small scale single task env. Im able to get around 50% for Cup to Drawer Task.

python scripts/gr00t_finetune.py \
--dataset-path datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim/gr1_arms_waist.CupToDrawer \
--num-gpus 2 --batch-size 25  --output-dir ~/checkpoints/24dc-cup-drawer-09jul-tune-visual-2gpu  \
--data-config fourier_gr1_arms_waist --max-steps 16000 \
--embodiment_tag gr1 --save-steps 2000 --tune-visual

python3 scripts/simulation_service.py --client     \
--env_name gr1_unified/PnPCupToDrawerClose_GR1ArmsAndWaistFourierHands_Env  \
   --video_dir ./videos4     --max_episode_steps 720     --n_envs 5     --n_episodes 10

....
Collecting 10 episodes took 99.85 seconds
Results for gr1_unified/PnPCupToDrawerClose_GR1ArmsAndWaistFourierHands_Env:
Success rate: 0.50

Note i use --tune-visual in this case.

193e6871-aba9-4951-8a6c-be189841dbe6_success1.mp4

Checking out others when i have more time soon

[4133223]

@youliangtan Thank you for your reply. Currently, I am also experimenting with the --tune-visual parameter. However, this has led to some issues with multi-GPU training, so I am now using single-GPU training and hoping for good results.

[youliangtan]

@4133223 i have a temp fix for multigpu visual tuning here: #257

[4133223]

Thank you. I will start multi-GPU training right away.

[youliangtan]

Also noted that it is important to run the with more number of episodes during rollout eval n_episodes. I tried with just 10 initially and the variance is quite high, so run it with 30 now.

python scripts/gr00t_finetune.py --dataset-path /home/youliang/gear/clean_repo/Isaac-GR00T/datasets/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim/gr1_arms_waist.CuttingboardToPan --num-gpus 2 --batch-size 25  --output-dir ~/checkpoints/24dc-board-pan-10jul-tune-visual-2gpu  --data-config fourier_gr1_arms_waist --max-steps 10000 --embodiment_tag gr1 --save-steps 2000 --tune-visual

# SR = 63% 
python3 scripts/simulation_service.py --client   \
  --env_name gr1_unified/PosttrainPnPNovelFromCuttingboardToPanSplitA_GR1ArmsAndWaistFourierHands_Env \
  --video_dir ./videos-b2p \
  --max_episode_steps 720 \
  --n_envs 5 \
  --n_episodes 30

[4133223]

@youliangtan Thank you for your reminder. I'm wondering, have you trained the model on only a single task, or have you conducted training across multiple tasks to verify the model's generalization capability? If the latter, what were the test results?