Improve RubyLLM::Chat#with_params (#265) by allowing to override default params

[MichaelHoste]

What this does

This PR slightly improves what #265 does by allowing the override of default parameters.

I just inverted the order of the deep_merge between payload and params, refactored the Utils method to make it more generic and added an Anthropic test.

This small PR solves many issues that we encountered:

• max_tokens is mandatory in the Anthropic API, and this gem sets it as the maximum tokens allowed by the model, but sometimes we want to force a lower value.

• We use this gem for Mistral by setting openai_api_base as https://api.mistral.ai/v1 because it's almost compatible. The only issue we encountered was that Mistral doesn't support the stream_options: { include_usage: true } parameter that is always added to the request for OpenAI. Now we can just add .with_params(stream_options: nil) to make it compatible.

• payload.compact! also clean the extra param when we want to use the default model temperature with with_temperature(nil)

My intuition is that it could help other people fix similar situations.

@compumike suggested this, but I feel that this PR is even simpler:

Future proposal: fully general block passing

It's possible that in addition to this #with_options, we may want the ability to pass in a block that allows modifying the payload after it's deep_merged but before it's actually sent. For example, this might look like:

chat = RubyLLM
  .chat(model: "gpt-4o-search-preview", provider: :openai)
  .with_options(web_search_options: {search_context_size: "medium"}) do |payload|
    payload.delete(:temperature)
 end
chat.ask(...)

Not necessary in this case, since with_temperature(nil) works. But allows future customization for power users.

Type of change

• Bug fix
• New feature
• Breaking change
• Documentation
• Performance improvement

Scope check

• I read the Contributing Guide
• This aligns with RubyLLM's focus on LLM communication
• This isn't application-specific logic that belongs in user code
• This benefits most users, not just my specific use case

Quality check

• I ran overcommit --install and all hooks pass
• I tested my changes thoroughly
• I updated documentation if needed
• I didn't modify auto-generated files manually (models.json, aliases.json)

API changes

• Breaking change
• New public methods/classes
• Changed method signatures
• No API changes

Related issues

Related to #265

[compumike]

@MichaelHoste I agree that it would be great to have some way to "dangerously" mutate the params after the payload!

FYI the merge order (merge payload after params) I implemented in #265 came directly due to @crmne 's comment #130 (review) on an earlier PR that had it like yours (merge params after payload):

I like the direction, but let's fix the merge order:

render_payload(...).merge(options) is dangerous because user options could override critical RubyLLM parameters like model, messages, stream, etc.

Simple fix: Change it to options.merge(render_payload(...)) instead. This way user options go in first, then RubyLLM's required parameters take precedence.

So I think any proposed solution here has to address that comment if it's going to get merged.

My hope with proposing the block format:

chat = RubyLLM
  .chat(model: "gpt-4o-search-preview", provider: :openai)
  .with_params(web_search_options: {search_context_size: "medium"}) do |payload|
    payload.delete(:temperature)
 end
chat.ask(...)

is that the block style would be considered to be a "power user feature" enough that it's not an easy footgun, compared to just passing in keyword arguments.

Another alternative might be that #with_params remains the Chat-level defaults that are persisted and applied before any individual message payload, but that an individual #complete call might take some extra argument, for example.

But I'd love to see other variations. There may be a much simpler way that makes everybody happy here. 😄

[compumike]

The hacky, monkeypatch way to set max_tokens right now:

context = RubyLLM.context # Important: must use a new context to ensure a separate RubyLLM::Connection is used for this Chat!
chat = RubyLLM
  .chat(model: "claude-3-5-haiku-20241022", provider: :anthropic, context: context)

### BEGIN INSTANCE MONKEYPATCH
class << chat
  attr_reader :connection
end

class << chat.connection
  attr_reader :responses
  
  def post(url, payload, &)
    puts "payload before: #{payload.inspect}"
    payload[:max_tokens] = 4
    puts "payload after: #{payload.inspect}"

    super(url, payload, &)
  end
end
### END INSTANCE MONKEYPATCH

chat.ask("What is the capital of France?")

payload before: {model: "claude-3-5-haiku-20241022", messages: [{role: "user", content: [{type: "text", text: "What is the capital of France?"}]}], temperature: 0.7, stream: false, max_tokens: 8192}
payload after: {model: "claude-3-5-haiku-20241022", messages: [{role: "user", content: [{type: "text", text: "What is the capital of France?"}]}], temperature: 0.7, stream: false, max_tokens: 4}
=> #<RubyLLM::Message:0x000074ea7f5682c0
 @content=#<RubyLLM::Content:0x000074ea8005ce38 @attachments=[], @text="The capital of France">,
 @input_tokens=14,
 @model_id="claude-3-5-haiku-20241022",
 @output_tokens=4,
 @role=:assistant,
 @tool_call_id=nil,
 @tool_calls=nil>

[MichaelHoste]

Thanks for this precision @compumike, i completely missed it when reading your PR, sorry about that!

I have mixed feelings about the "block" approach. My main issue is that for some models, this would work:

RubyLLM
  .chat(model: "gpt-4o-mini-2024-07-18", provider: :openai)
  .with_params(max_tokens: 40) # or even max_completion_tokens

But with others, where max_params is already in the payload, it won't work and this code will be needed:

RubyLLM
  .chat(model: "claude-3-5-haiku-20241022", provider: :anthropic)
  .with_params do |payload|
    payload.merge!(max_tokens: 40)
 end

Since the user doesn't know exactly what is in the final payload, the behavior could be unexpected for other similar scenarios.

Since there are very few parameters that should be protected at any cost. Wouldn't a more straightforward solution like this be preferable to avoid any footguns?

PROTECTED_PARAMS = %w(model tools messages stream) # maybe more?

protected_params = params.keys.map(&:to_s) & PROTECTED_PARAMS
raise ArgumentError, "Protected parameters: #{protected_params.join(', ')}" unless present_params.empty?

[compumike]

The PROTECTED_PARAMS approach could work for me.

(I do wonder if there are any use cases where it should be applied in a "deep" way to protect payload.dig("a", "b") while allowing someone to set payload.dig("a", "c"), but I can't think of any off the top of my head.)

[Aesthetikx]

Perhaps this is for a different topic, but locally I made the following change:

  module RubyLLM
    module Utils
      # Overridden to merge arrays, so that with_params can provide additional tools.
      def self.deep_merge(params, payload)
        params.merge(payload) do |_key, params_value, payload_value|
          if params_value.is_a?(Hash) && payload_value.is_a?(Hash)
            deep_merge(params_value, payload_value)
          elsif params_value.is_a?(Array) && payload_value.is_a?(Array)
            params_value + payload_value
          else
            payload_value
          end
        end
      end
    end
  end

The changed part is the support for merging array values. This was because you cannot currently use #with_tool(s) and simultaneously add to tools:, as the value specified in with_params will be replaced with the one provided by the library. I think it is debatable if you would want your user provided array to replace, be replaced, or be merged with the library provided values, but this was something I had to do to enable some tools provided by Anthropic in addition to my own.