Skip to content

GH-3403: Add caching for JSON schema generation in function calling #3862

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

Seol-JY
Copy link
Contributor

@Seol-JY Seol-JY commented Jul 20, 2025

Fixes GH-3403

Add ConcurrentHashMap-based caching to eliminate repeated schema generation for identical method signatures in tool invocations.

  • Cache tool definitions by Method instance in ToolDefinitions
  • Cache JSON schemas with method signature + options as key
  • Add comprehensive test coverage for caching behavior

Performance improvement: Eliminates CPU overhead from repeated schema generation in function calling applications.

…ction calling

Fixes spring-projectsGH-3403 (spring-projects#3403)

Add ConcurrentHashMap-based caching to eliminate repeated schema
generation for identical method signatures in tool invocations.

* Cache tool definitions by Method instance in ToolDefinitions
* Cache JSON schemas with method signature + options as key
* Include comprehensive test coverage for caching behavior

Performance improvement: Eliminates CPU overhead from repeated
schema generation in function calling applications.

Signed-off-by: Seol-JY <[email protected]>
Fix formatting violations detected by spring-javaformat-maven-plugin

Signed-off-by: Seol-JY <[email protected]>
@YunKuiLu
Copy link
Contributor

Can we also cache ToolCallback[] in MethodToolCallbackProvider ? 🤔

Such as:

public final class MethodToolCallbackProvider implements ToolCallbackProvider {
    @Override
    public ToolCallback[] getToolCallbacks() {
    if (this.cachedToolCallbacks == null) {
	    synchronized (this) {
		    if (this.cachedToolCallbacks == null) {
			    var toolCallbacks = this.toolObjects.stream()
				    .map(toolObject -> Stream
					    .of(ReflectionUtils.getDeclaredMethods(AopUtils.isAopProxy(toolObject)
							    ? AopUtils.getTargetClass(toolObject) : toolObject.getClass()))
					    .filter(this::isToolAnnotatedMethod)
					    .filter(toolMethod -> !isFunctionalType(toolMethod))
					    .filter(ReflectionUtils.USER_DECLARED_METHODS::matches)
					    .map(toolMethod -> MethodToolCallback.builder()
						    .toolDefinition(ToolDefinitions.from(toolMethod))
						    .toolMetadata(ToolMetadata.from(toolMethod))
						    .toolMethod(toolMethod)
						    .toolObject(toolObject)
						    .toolCallResultConverter(ToolUtils.getToolCallResultConverter(toolMethod))
						    .build())
					    .toArray(ToolCallback[]::new))
				    .flatMap(Stream::of)
				    .toArray(ToolCallback[]::new);
    
			    validateToolCallbacks(toolCallbacks);
    
			    this.cachedToolCallbacks = toolCallbacks;
		    }
	    }
    }
    return this.cachedToolCallbacks;
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Consider caching JSON schema generation in function calling
2 participants