Missing custom dreamerv3 configs and ROS 2 environment wrapper

[k-iswar]

Hi Jack,

First off, thank you for putting together this repository and leaving such detailed notes! I’ve successfully built the ROS 2 workspace, got the 4WD rover spawning perfectly in Gazebo, and set up the jaxRos conda environment following your guide.

However, I hit a roadblock when trying to start the Dreamerv3 training. It looks like your custom dreamerv3 code/folder wasn't committed to the repository.

To try and bypass it, I cloned the official danijar/dreamerv3 repo into the workspace and ran your exact command from the notes:

env LD_LIBRARY_PATH="$FILTERED_LD_LIBRARY_PATH" CUDA_HOME="" XLA_PYTHON_CLIENT_PREALLOCATE=false XLA_PYTHON_CLIENT_ALLOCATOR=platform python dreamerv3/dreamerv3/main.py --configs rover --logdir ~/logdir/rover_training_1

As expected, it throws a KeyError: 'rover' because the official repository doesn't have your custom rover settings in configs.yaml.

More importantly, I realized that I am likely missing the custom environment "glue" code you wrote that actually bridges the Dreamerv3 agent to the Gazebo simulation (subscribing to the ROS 2 /camera topics and publishing to /cmd_vel).

When you have a moment, could you please upload the custom Dreamerv3 configurations and the ROS 2 wrapper code you used? I would love to get this training pipeline fully running.

Thanks again for your hard work on this project!

[jackvice]

k-iswar, here is the dreamerV3 repo.
https://github.com/jackvice/dreamerJMV3
The leorover.py is currently configured for Active Vision but you will see some other environments as well. I will be cleaning up the repo and updating the readme early next week.

[k-iswar]

k-iswar, here is the dreamerV3 repo. https://github.com/jackvice/dreamerJMV3 The leorover.py is currently configured for Active Vision but you will see some other environments as well. I will be cleaning up the repo and updating the readme early next week.

Great to hear it from you. Can you give the detailed structure and steps to run your repo with each module, at my system? It would be of great help to me.

[jackvice]

Yes, I can give you the current setup. I'll make a seperate readme tomorrow or wednesday. The full integrated pipeline also uses the attention repo:
https://github.com/jackvice/attention
The full pipeline is very compute intensive. I am running on GTX-4090.

[k-iswar]

Yes, I can give you the current setup. I'll make a seperate readme tomorrow or wednesday. The full integrated pipeline also uses the attention repo: https://github.com/jackvice/attention The full pipeline is very compute intensive. I am running on GTX-4090.

Hello Jack. You might be busy but we are eagerly waiting for your update on the repo. We are looking forward to run and test your thing.

[jackvice]

I pushed an Active_Vision_README.md to /Roboterrain which explains how to run integrated attention mechanism and DreamV3 with Roboterrain. Also see preprint for description:
https://www.preprints.org/manuscript/202604.0068

[k-iswar]

I pushed an Active_Vision_README.md to /Roboterrain which explains how to run integrated attention mechanism and DreamV3 with Roboterrain. Also see preprint for description: https://www.preprints.org/manuscript/202604.0068

Thanks for this much detailed read file. I tried the repo 3 days back, got some errors in my terminal but i solved those. But now i am stuck with the trained model. You have used trained model named checkpoint_epoch_1000.pkl but it is missing from the repo. I tried to train new model with SIT dataset, that is very massive around 100GB of dataset. It would be of great help if you could share your trained model through drive link or whichever way you like.

[jackvice]

Yea, I forgot to push the model. I just pushed it now /model_output/checkpoint_epoch_1000.pkl. Let me know if any other trouble.

[k-iswar]

Yea, I forgot to push the model. I just pushed it now /model_output/checkpoint_epoch_1000.pkl. Let me know if any other trouble.

Hey jackvice,

Thanks for the quick update!

I just checked, but it looks like the checkpoint_epoch_1000.pkl file still didn't make it to the repository. I noticed the .gitignore was the only file updated in that last commit, so it seems like your .gitignore rules might be actively blocking .pkl files or the model_output/ directory from being staged.

If the file is under 100MB, you might just need to force add it using git add -f /model_output/checkpoint_epoch_1000.pkl.

However, if the checkpoint is larger than 100MB, GitHub's file size limit will probably block the push anyway. If that's the case, uploading the .pkl to GitHub Releases or sharing a quick Google Drive link might be the easiest workaround.

[jackvice]

Yea, it was *.pkl in gitignore. it's pushed up now:
https://github.com/jackvice/attention/tree/main/model_output