Skip to content

docs: rai simbench docs update #665

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 8 commits into
base: development
Choose a base branch
from

Conversation

jmatejcz
Copy link
Contributor

@jmatejcz jmatejcz commented Aug 6, 2025

Purpose

Fixing couple small issues that were occuring during execution:

  • Some scenarios didn't have proper level assigned
  • Errors were catched in a manner that was not informative and lead to further execution
  • Fixed response returned by MoveObjectFromToTool

Proposed Changes

  • Now all scenarios have proper level assignment in predefined scenarios
  • Errors are now catched in test_models function instead of in run_next. Before that when an unexpected exception occured, it would affect the single scenario execution. After that the next scenario would execute as planned, but that was the problem, because usually the problem was outside of scenario scope and would reoccur in every scenario till the end - for example cuda OOM. In such case it was a wasted time and the avg scores were impared by the 0.0 scores that were result of error.
    Now when unexpected error occurs the whole benchmark will stop, preserving the scores that were achieved till this moment.

Issues

Testing

Just run manipulation benchmark

python src/rai_bench/rai_bench/examples/benchmarking_models.py

you can check out the logs to see that levels are assigned properly

@jmatejcz jmatejcz marked this pull request as ready for review August 6, 2025 10:44
@jmatejcz jmatejcz changed the title docs: update rai simbench docs update docs: rai simbench docs update Aug 6, 2025
@Juliaj
Copy link
Collaborator

Juliaj commented Aug 8, 2025

@jmatejcz , could this PR use a rebase ?

@jmatejcz jmatejcz force-pushed the jm/docs/rai-simbench-docs-update branch from 1b6618d to 98b4ec8 Compare August 11, 2025 16:12
@jmatejcz
Copy link
Contributor Author

@jmatejcz , could this PR use a rebase ?

it is fine now, but i removed poetry lock which was accidentally added add looked like a lot of lines changed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants