Skip to content

fix: manipulaiton bench fixes #653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: development
Choose a base branch
from
Open

fix: manipulaiton bench fixes #653

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d44d45
refactor: adjustment of error catching
jmatejcz Jul 25, 2025
6d37658
fix: response fix in MoveFromTo Tool
jmatejcz Jul 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions src/rai_bench/rai_bench/manipulation_o3de/benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,8 +308,7 @@ def run_next(self, agent: CompiledStateGraph, experiment_id: uuid.UUID) -> None:
self.logger.error(msg=f"Task timeout: {e}")
except GraphRecursionError as e:
self.logger.error(msg=f"Reached recursion limit {e}")
except Exception as e:
self.logger.error(msg=f"Unexpected errot occured: {e}")

te = time.perf_counter()
try:
score = scenario.task.calculate_score(self.simulation_bridge)
Expand Down
9 changes: 7 additions & 2 deletions src/rai_bench/rai_bench/test_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,12 @@ def test_models(
bench_logger=bench_logger,
)
except Exception as e:
bench_logger.critical(f"BENCHMARK RUN FAILED: {e}")
import traceback

bench_logger.critical(
f"{bench_conf.name} benchmark for {model_name}, vendor: {vendors[i]}, execution number: {u + 1}"
f"{bench_conf.name} benchmark for {model_name}, vendor: {vendors[i]}, repeat number: {u + 1}"
)
bench_logger.critical(f"BENCHMARK RUN FAILED: {e}")
error_msg = traceback.format_exc()
bench_logger.critical(error_msg)
print(error_msg)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding "Now when unexpected error occurs the whole benchmark will stop, preserving the scores that were achieved till this moment" - the exception handling here only catches, logs, and continues, correct? I'm trying to understand how the new changes will actually stop the benchmark run.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because the error was caught here: https://github.com/RobotecAI/rai/pull/653/files#diff-76d8a11a893998856be6b462a5f01a0f7c0d3a0013fb979e286fce9fc3618247L311

which led to continuing with other scenarios

3 changes: 1 addition & 2 deletions src/rai_bench/rai_bench/tool_calling_agent/benchmark.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -148,8 +148,7 @@ def run_next(self, agent: CompiledStateGraph, experiment_id: uuid.UUID) -> None:
self.logger.error(msg=f"Task timeout: {e}")
except GraphRecursionError as e:
self.logger.error(msg=f"Reached recursion limit {e}")
except Exception as e:
self.logger.error(msg=f"Unexpected error occured: {e}")

tool_calls = task.get_tool_calls_from_messages(messages=messages)
score = task.validate(tool_calls=tool_calls)
te = time.perf_counter()
Expand Down
6 changes: 3 additions & 3 deletions src/rai_core/rai/tools/ros2/manipulation/custom.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -245,12 +245,12 @@ def _run(
response = get_future_result(future, timeout_sec=20.0)

if response is None:
return f"Service call failed for point ({x:.2f}, {y:.2f}, {z:.2f})."
return f"Service call failed for point ({x1:.2f}, {y1:.2f}, {z1:.2f})."

if response.success:
return f"End effector successfully positioned at coordinates ({x:.2f}, {y:.2f}, {z:.2f}). Note: The status of object interaction (grab/drop) is not confirmed by this movement."
return f"End effector successfully positioned at coordinates ({x1:.2f}, {y1:.2f}, {z1:.2f}). Note: The status of object interaction (grab/drop) is not confirmed by this movement."
else:
return f"Failed to position end effector at coordinates ({x:.2f}, {y:.2f}, {z:.2f})."
return f"Failed to position end effector at coordinates ({x1:.2f}, {y1:.2f}, {z1:.2f})."


class GetObjectPositionsToolInput(BaseModel):
Expand Down
Loading