How can I capture or listen to the event when a tool is actually invoked? I would like to wrap some custom tags or markers before and after the tool execution.

[Jisprincess]

How can I capture or listen to the event when a tool is actually invoked? I would like to wrap some custom tags or markers before and after the tool execution.

such as :

findPeopleBen

[Jisprincess]

For example, I want to wrap the tool’s name, input, and output with tags. Since some tools take a long time to execute, having these tags would allow me to detect when a tool is being used and implement a waiting behavior accordingly — similar to how tool events work in CrewAI.

[YunKuiLu]

You can try my solution.

Inject your custom ObservationHandler Bean into your app.

@Configuration
public class MyObservationConfig {

    @Bean
    ToolExecuteObservationHandler toolExecuteObservationHandler() {
        return new ToolExecuteObservationHandler(List.of(ToolCallingObservationContext.class));
    }


    @Slf4j
    public static class ToolExecuteObservationHandler implements ObservationHandler {

        private final List<Class<? extends Observation.Context>> supportedContextTypes;

        public ToolExecuteObservationHandler(List<Class<? extends Observation.Context>> supportedContextTypes) {

            Assert.notNull(supportedContextTypes, "SupportedContextTypes must not be null");

            this.supportedContextTypes = supportedContextTypes;
        }

        @Override
        public boolean supportsContext(Observation.Context context) {
            return (context == null) ? false : this.supportedContextTypes.stream().anyMatch(clz -> clz.isInstance(context));
        }

        @Override
        public void onStart(Observation.Context context) {
            log.info("onStart: {}", context);
        }

        @Override
        public void onEvent(Observation.Event event, Observation.Context context) {
            log.info("onEvent: {} {}", event, context);
        }

        @Override
        public void onStop(Observation.Context context) {
            log.info("onStop: {}", context);
        }

        @Override
        public void onError(Observation.Context context) {
            log.error("onError: {}", context);
        }

    }

}

Now, you will receive onStart, onStop and other notifications when the tool is executed.

[Jisprincess]

@YunKuiLu
Thank you very much, but what I want is for the model tool invocation to be shown in the stream output, with tool usage wrapped in tags indicating the start and end of the invocation. I don't want it done via prompt constraints, because prompt constraints only summarize after the tool is finished — they don't reflect the tool invocation process in real time.

Flux<String> callTools = clientIng.build() .prompt("what today weather is ?") .stream() .content();

Output example:
hello <tool><tool name>query weather</tool name><tool content>rain</tool content></tool> weather is rain today

[YunKuiLu]

Sorry, I misunderstood your question.

You can try User-Controlled Tool Execution, although it might be a bit of a hassle and require some hands-on coding. (This document uses ChatModel as an example, rather than ChatClient, so you may need to make some adjustments.)

Hi @markpollack @ilayaperumalg. Is there a better approach to handle this case?

[YunKuiLu]

@markpollack @ilayaperumalg
I tried using ChatClient for "User-Controlled Tool Execution", and found it difficult. At least it requires writing a lot of code.😥

1. The executeToolCalls method of ToolCallingManager requires a Prompt as a parameter, but there's no way for me to obtain it from ChatClient (which means I have to implement the tool execution logic myself).
2. There is no simple method for me to get the result of tool execution from ToolExecutionResult (I need to iterate through it).

public Flux<String> tool2() {
    Flux<ChatClientResponse> response = chatClient.prompt()
            .options(ToolCallingChatOptions.builder().internalToolExecutionEnabled(false).build())
            .tools(this)
            .user("What is the weather in New York?")
            .stream()
            .chatClientResponse();

    return response.flatMap(chatClientResponse -> {
        ChatResponse chatResponse = chatClientResponse.chatResponse();
        if (chatResponse.hasToolCalls()) {
            Prompt prompt = null; // TODO I can't get Prompt from ChatClientResponse
            ToolExecutionResult toolExecutionResult = toolCallingManager.executeToolCalls(prompt, chatResponse);
            // TODO I need tool execution result but I can't get it easily
        }
        return Flux.just(chatResponse.getResult().getOutput().getText());
    });
}

[Jisprincess]

thanks ,but i thinnk it not down my question。for example，many tools need to do 。 i wand get one result ,but need wait all tool finish。one tool lable can not show in Flex in time.

[Jisprincess]

@YunKuiLu

[Jisprincess]

In the default implementation, does it only start outputting the streaming response after all tool executions are completed and the response from the stream interface no longer requires any tool calls?

[markpollack]

Hi @YunKuiLu - yea the answer to the question 'what tools were called' seems to be a fairly common one and I agree that the 'user controlled tool execution' gives you the raw hook to implement this, but it isn't a trivial task. Also I think @Jisprincess is asking for real time info on tool execution to be streamed back? I'll reopen this issue , as I think the observability solution is pretty lightweight and will meet the need of many users. also, it would be good to verify the user controlled tool execution in this context. Then we can create a documentation issue or perhaps an improvment to collect this informaiton.