How do I set maxTokens?

[rungekutta1942]

Hi, how do I override maxTokens from default? I try setting in a ProcessOptions like this:

ProcessOptions processOptions = ProcessOptions.DEFAULT.withBudget(Budget.DEFAULT.withTokens(15000000));

... however at runtime, log files still show the execution runs with defaults.

The ProcessOptions.Builder stuff doesn't seem to be available from Java?

[johnsonr]

Set it on the LlmOptions in your PromptRunner.

context.ai()
                .withLlm(myLlm.withMaxTokens(MAX_TOKENS))

You can also create and reuse an LlmOptions object.

You can also externalize it using ConfigurationProperties and set it in application.yml like this:

  chat-llm:
    model: gpt-4.1-mini
    temperature: 0.0
    max-tokens: 200000

See https://github.com/embabel/impromptu/blob/main/src/main/java/com/embabel/impromptu/ImpromptuProperties.java for an example. Impromptu is completely Java.

[rungekutta1942]

Thank you!