86: Add default limit for tools completions #87

rhys117 · 2025-04-01T13:20:25Z

Resolves #86

Purpose

Introduces a configurable limit on tool completions to prevent infinite loops and excessive API usage. This feature adds protection against scenarios where AI responses might trigger continuous tool calls.

Implementation Details

Added max_tool_llm_calls configuration option (default: 25 calls)
Per-chat override capability through existing contexts
Implemented ToolCallLimitReachedError when limit is exceeded
Added tracking of tool llm calls via number_of_tool_completions counter
Can be overridden for unlimited tool completions with nil

Usage Example

# Global configuration
RubyLLM.configure do |config|
  config.max_tool_llm_calls = 10  # Set default limit
end

Testing

Added RSpec test cases for OpenAI + Claude + Openrouter
Verified both global configuration and per-chat override functionality
Confirmed error raised when limit is reached

Documentation

Added section in tools.md guide explaining the feature
Included examples for both global and per-chat configuration

TODO

Add VCR cassettes for testing with additional providers (I do not have API keys for these providers)

rhys117 · 2025-04-01T13:24:31Z

lib/ruby_llm/chat.rb

@@ -105,6 +107,10 @@ def handle_tool_calls(response, &)
    end

    def execute_tool(tool_call)
+      raise ToolCallsLimitReachedError, "Tool calls limit reached: #{@max_tool_calls}" if max_tool_calls_reached?


It might be worth discussing if this should be handled via the chat instead of raising an unhandled error.

- Otherwise the chat object could not have 'ask' executed on again due to malformed messages

crmne

changing the whole interface to chat is a bit heavy handed. this should be a simple config change.

crmne · 2025-04-23T20:24:15Z

lib/ruby_llm.rb

-    def chat(model: nil, provider: nil)
-      Chat.new(model: model, provider: provider)
+    def chat(model: nil, provider: nil, max_tool_completions: config.max_tool_completions)
+      Chat.new(model: model, provider: provider, max_tool_completions: max_tool_completions)


changing the whole interface to chat is a bit heavy handed. this should be a simple config change.

Thanks for the feedback @crmne. I've adjusted things so the chat interface isn't modified, and instead, a single instance can use an override from the config using with_max_tool_completions.

Please let me know what you think?

@crmne would love an update here ... I'm getting looping tool calls as well, and would like to avoid implementing my own solution.

crmne

Thank you for your work!

Left you some comments + I'm not a big fan of the naming. max_tool_calls is less wordy. Same thing for all the instances of the name, like the error name and the documentation.

crmne · 2025-06-10T11:25:09Z

spec/ruby_llm/chat_tools_spec.rb

+          { title: Faker::Name.name, score: rand(1000) },
+          { title: Faker::Name.name, score: rand(1000) },
+          { title: Faker::Name.name, score: rand(1000) }


I think we can come up with a fake title instead of adding a dependency for these three lines of code.

crmne · 2025-06-10T11:25:51Z

lib/ruby_llm/active_record/acts_as.rb

+      def with_max_tool_completions(...)
+        to_llm.with_max_tool_completions(...)
+        self
+      end
+


We now have configuration Contexts so we don't need the per-chat max tool completions method.

crmne · 2025-06-10T11:26:52Z

lib/ruby_llm/chat.rb

@@ -11,7 +11,7 @@ module RubyLLM
  class Chat # rubocop:disable Metrics/ClassLength
    include Enumerable

-    attr_reader :model, :messages, :tools
+    attr_reader :model, :messages, :tools, :number_of_tool_completions


No need to have it as attr_reader.

crmne · 2025-06-10T11:27:22Z

lib/ruby_llm/chat.rb

+    def with_max_tool_completions(max_tool_completions)
+      @max_tool_completions = max_tool_completions
+      self
+    end
+


We now have configuration Contexts so we don't need the per-chat max tool completions method.

👌 Much nicer for overridding

crmne · 2025-06-10T11:39:26Z

lib/ruby_llm/chat.rb

+      if max_tool_completions_reached?
+        raise ToolCallCompletionsLimitReachedError, "Tool completions limit reached: #{@max_tool_completions}"
+      end
+
+      @number_of_tool_completions += 1


Shouldn't this be at the top of the method? Say max_tool_completions is 0, that would mean that we should process 0 tool completions, right?

Thanks for the review @crmne, you're right, this should have been before the max_tool_completions_reached? check. I've amended this now along with your other feedback.

I haven't manually tested the new changes yet, but the test cases are passing for the providers I have keys for.

Please let me know what you think.

rhys117 · 2025-06-12T12:52:50Z

lib/ruby_llm/chat.rb

+      if context.config
+        @config = context.config
+        @max_tool_llm_calls = @config.max_tool_llm_calls
+      end


Wrapped this in a conditional to ensure that the config was present on the context. This more closely aligns with the assignment in the initialiser (@config = context&.config || RubyLLM.config)|

If I've made a mistake here, please let me know.

viktorianer · 2025-07-26T13:23:11Z

lib/ruby_llm/configuration.rb

@@ -54,6 +56,9 @@ def initialize
      @default_embedding_model = 'text-embedding-3-small'
      @default_image_model = 'dall-e-3'

+      # Default restrictions
+      @max_tool_llm_calls = 25


Isn't 25 a bit too high? I'm thinking of a case where you'd need 25 attempts before succeeding. Wouldn't 5 be sufficient?

Personally, I'd err on the side of caution and allow a higher limit, seeing as these changes will affect existing implementations if not released with a major version bump.

That said, I'd be overriding this with something more conservative myself though.

I think we can—and should—limit it to what’s appropriate and what we consider a sensible default.

In my case, I even use a max limit of 3.

I believe that in most scenarios, the limit will lean toward the lower side. But I could be wrong.

If we provide an option to override with a higher limit when needed (e.g., via ask), then a low default should be sufficient.

If these changes were only counting consecutive errors, I'd agree with you here.

However, this implementation counts all LLM provider calls regardless of success or error. I chose this approach to prevent runaway loops where tools return valid responses but trigger unintended cascading calls, like a search tool that keeps finding "relevant" results that spawn more searches.

tpaulshippy · 2025-07-26T19:20:32Z

Is this maximum scoped to a particular tool/call or global for all tool calls? We have situations where we design the prompt to call a tool 5-10 times so we can get a bunch of different things done in one request. Would this limit apply in those situations? Is there any way to get the maximum to only apply after it starts looping?

rhys117 · 2025-07-27T00:46:59Z

Is this maximum scoped to a particular tool/call or global for all tool calls? We have situations where we design the prompt to call a tool 5-10 times so we can get a bunch of different things done in one request. Would this limit apply in those situations? Is there any way to get the maximum to only apply after it starts looping?

I've designed it to limit the number of turns with the LLM provider per '.ask' call (turns are referred to as 'completions' throughout this merge request and the repository). I've included some documentation as part of these changes -
https://github.com/crmne/ruby_llm/blob/397438e096358bbca8ad28c3a3fb9aa31b4ec5e7/docs/guides/tools.md#maximum-tool-llm-requests

Please let me know if you think the docs could be clearer. It's always good to see it from someone else's perspective!

tpaulshippy · 2025-07-27T02:05:26Z

Ok makes sense. Would be a cool feature to just limit the endless looping though.

rhys117 · 2025-07-27T02:37:03Z

Ok makes sense. Would be a cool feature to just limit the endless looping though.

Out of curiosity, for my understanding, do you have ruby code that loops rather than back and forth requests to the LLM provider?

tpaulshippy · 2025-07-27T02:43:44Z

Ok makes sense. Would be a cool feature to just limit the endless looping though.

Out of curiosity, for my understanding, do you have ruby code that loops rather than back and forth requests to the LLM provider?

I'm talking about situations where the tool call fails but not in a fatal way, like a validation error. In these cases, we make the execute method return a hash with an error key. The library sends the validation error back to the LLM and it just keeps doing it over and over.

tpaulshippy · 2025-07-27T02:52:10Z

See https://rubyllm.com/guides/error-handling#handling-errors-within-tools

rhys117 · 2025-07-27T04:15:11Z

See https://rubyllm.com/guides/error-handling#handling-errors-within-tools

Got it, this should work to prevent those loops if they're done in a single 'ask' call.

Are you suggesting there be an option to only count the tools that 'error' when counting the number of turns?

tpaulshippy · 2025-07-27T04:17:41Z

See https://rubyllm.com/guides/error-handling#handling-errors-within-tools

Got it, this should work to prevent those loops if they're done in a single 'ask' call.

Are you suggesting there be an option to only count the tools that 'error' when counting the number of turns?

Yeah that might do it, especially if it were scoped by tool. So tool A could have a max error count of X and tool B could have a max error count of Y.

rhys117 added 3 commits April 1, 2025 22:54

feat: add limit to tools calls

e7f7a0f

docs: add docs for removing limit

47c8742

docs: Fix docs

190e9de

rhys117 commented Apr 1, 2025

View reviewed changes

rhys117 added 2 commits April 1, 2025 23:30

docs: improve doc - more accurate & organisation

fe837de

bug: reset tools calls for each new 'ask' call

0f7ce26

rhys117 marked this pull request as draft April 1, 2025 13:54

bug: move error so tool result is still added

6b807cf

- Otherwise the chat object could not have 'ask' executed on again due to malformed messages

crmne added the enhancement New feature or request label Apr 2, 2025

rhys117 added 8 commits April 3, 2025 20:53

merge: main

df12db9

chore: rework approach to rely on completions instead of tool calls

3af500d

bug: fix attr_reader declaration to match number_of_tool_completions

39894a8

test: add better example

4718e2d

docs: correct docs after changing naming/strategy

7885c4b

merge: Merge branch 'main' into max-tool-calls

585af65

docs: add default to docs

df26988

chore: rename error to match new naming

19c0cd1

rhys117 marked this pull request as ready for review April 7, 2025 12:15

rhys117 changed the title ~~86: Add default limit for tools calls~~ 86: Add default limit for tools completions Apr 7, 2025

crmne requested changes Apr 23, 2025

View reviewed changes

rhys117 added 7 commits April 24, 2025 21:25

Merge branch 'main' into max-tool-calls

3ad25f4

chore: reorder configuration accessor and add comment

d94d6dd

style: cops - disable / fix

71dba53

test: add spec for configured limit

6455f08

docs: adjust tools docs

5a44bff

merge: main

17b55cc

bug: ensure with_max_tool_completions available through acts_as helpers

d09ce92

rhys117 mentioned this pull request May 9, 2025

Looping Tool Calls #86

Open

crmne requested changes Jun 10, 2025

View reviewed changes

Merge branch 'main' into max-tool-calls

69af539

rhys117 added 9 commits June 11, 2025 23:18

deps: remove faker gem

018cc3b

chore: use existing context instead of with_max_tool_completions

4cb6765

test: adjust spec for context use

6cb8ec3

chore: rename to max_tool_llm_calls

4cdb924

docs: minor doc correction after rename

f8fc8a5

chore: rename error class

731fcc6

test: ensure spec cases for openrouter, openai, anthropic passing

092ff2c

bug: fix +1 issue for llm tool lomts

a0a1fa8

docs: improve tool docs for max tool llm calls

397438e

rhys117 commented Jun 12, 2025

View reviewed changes

rhys117 requested a review from crmne June 12, 2025 12:57

viktorianer reviewed Jul 26, 2025

View reviewed changes

Uh oh!

86: Add default limit for tools completions #87

Are you sure you want to change the base?

86: Add default limit for tools completions #87

Conversation

rhys117 commented Apr 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Implementation Details

Usage Example

Testing

Documentation

TODO

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhys117 Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tpaulshippy commented Jul 26, 2025

Uh oh!

rhys117 commented Jul 27, 2025

Uh oh!

tpaulshippy commented Jul 27, 2025

Uh oh!

rhys117 commented Jul 27, 2025

Uh oh!

tpaulshippy commented Jul 27, 2025

Uh oh!

tpaulshippy commented Jul 27, 2025

Uh oh!

rhys117 commented Jul 27, 2025

Uh oh!

tpaulshippy commented Jul 27, 2025

Uh oh!

Uh oh!

rhys117 commented Apr 1, 2025 •

edited

Loading

rhys117 Jun 12, 2025 •

edited

Loading