Unexpected garbled output when using LIBERO fine-tuned checkpoints

[SlenderMongoose]

Hi,
Thanks for the impressive work! I’ve been exploring the repository and successfully ran the Quick Interactive M1 Demo from the README. The model correctly identified the apple's position, and my visualization confirmed the accuracy.

Output: ['{"label": "apple", "bbox_2d": [304, 226, 345, 267]}']

Furthermore, when I fed it an image from the LIBERO evaluation environment, it could still correctly identify the position of the plate.

Output: ['{"label": "plate", "bbox_2d": [26, 147, 68, 179]}']

However, these are all based on the Pretrain-RT-1-Bridge checkpoint. When using the libero-related checkpoints, the model's output becomes "garbled" or nonsensical (unlike the clean results from the M1 demo).

Output: ['极为这般 ENCჭpunkt\n极为勍:\n桀chantment实效uri\n极力栽培差别澎湃\néra:\n粜acity!\n粜acity!\nerrer!\nerrer?\npragma:\nerrer?\n搛\n恪光阴;\n债权?\n\t \nrang!\n恪光阴;\n恪光阴!\n機能: []\nável!\n.pathname!\n機能:!\n恪光阴!\n.pathname!\n機能:!\n恪光阴!\n.pathname!\n恪光阴!\n機能:!\n恪光阴!\n.pathname!\n恪光阴!\n機能:!\n恪光阴!\n.pathname!\n恪光阴!\n機能:!\n恪光阴!\n.pathname!\n恪光阴!\n機能:!\n恪光阴!\n.pathname!\n恪光阴!\n']

My questions are:

1. Does the fine-tuned LIBERO model sacrifice its general grounding/VQA capability to achieve higher task success rates?
2. I noticed the Pretrain-RT-1-Bridge ckpt uses dinov2_vits14 while the libero-related ckpts use dinov2_vitl14. Why is that? What does "Pretrain" signify here? Is it a general-purpose robotic foundation model before tuning into libero ckpt? What is the relationship between these versions?
3. Since the Pretrain-RT-1-Bridge ckpt already demonstrates strong zero-shot grounding capabilities, I am curious about its potential for direct libero evaluation. Is that possible?

Please excuse any imprecise terminology, as I am still a newcomer to this field and am learning the ropes.

[MichaelYu781]

Hi,

Thanks for your interest and for the detailed observations! Here are the answers to your questions:

1. Regarding the loss of grounding/VQA capability: To ensure a fair comparison with other methods on the LIBERO benchmark, we did not include general VQA data during the fine-tuning process. This exclusion likely led to "catastrophic forgetting" of the conversational abilities, resulting in the garbled output you saw. In fact, we believe that co-training with general VQA data would actually further improve the task success rate, in addition to retaining those capabilities.

2. Regarding the architecture and "Pretrain" definition: Thank you for noticing the detail about the visual encoders. There is indeed a slight version misalignment between the checkpoints (ViT-S vs. ViT-L). However, based on our experience, the difference between the Small and Large versions has a negligible impact on the final results. As for the terminology, "Pretrain" here specifically refers to whether the VLM has been trained on general tasks using simulation data before any specific downstream tuning.

3. Regarding direct evaluation on LIBERO: While the Pretrain-RT-1-Bridge checkpoint does demonstrate strong generalization and zero-shot grounding (as seen in your visualization), its action head is likely not directly compatible with the LIBERO environment. Therefore, it cannot be used for direct evaluation without some form of adaptation or fine-tuning to align the action space.

I hope this clarifies things. Let me know if you have any other questions!

[SlenderMongoose]

Hi,

Thanks for your interest and for the detailed observations! Here are the answers to your questions:

1. Regarding the loss of grounding/VQA capability: To ensure a fair comparison with other methods on the LIBERO benchmark, we did not include general VQA data during the fine-tuning process. This exclusion likely led to "catastrophic forgetting" of the conversational abilities, resulting in the garbled output you saw. In fact, we believe that co-training with general VQA data would actually further improve the task success rate, in addition to retaining those capabilities.
2. Regarding the architecture and "Pretrain" definition: Thank you for noticing the detail about the visual encoders. There is indeed a slight version misalignment between the checkpoints (ViT-S vs. ViT-L). However, based on our experience, the difference between the Small and Large versions has a negligible impact on the final results. As for the terminology, "Pretrain" here specifically refers to whether the VLM has been trained on general tasks using simulation data before any specific downstream tuning.
3. Regarding direct evaluation on LIBERO: While the Pretrain-RT-1-Bridge checkpoint does demonstrate strong generalization and zero-shot grounding (as seen in your visualization), its action head is likely not directly compatible with the LIBERO environment. Therefore, it cannot be used for direct evaluation without some form of adaptation or fine-tuning to align the action space.

I hope this clarifies things. Let me know if you have any other questions!

Thanks for the patient explanation! This cleared things up for me. I really appreciate your help.