feat: Add utilities for converting tools into human-in-the-loop tools

[sjrl]

Related Issues

• fixes https://github.com/deepset-ai/haystack-private/issues/186

Proposed Changes:

This PR introduces a human-in-the-loop (HITL) mechanism for tool execution in Haystack, allowing users to confirm, reject, or modify tool invocations interactively.

• haystack_experimental/tools/human_in_the_loop.py

  • Implements ConfirmationResult dataclass to capture user decisions.
  • Adds RichConsolePrompt and SimpleInputPrompt for interactive confirmation via Rich or standard input.
  • Defines DefaultPolicy and AutoConfirmPolicy for handling user responses.
  • Provides confirmation_wrapper to wrap any tool with HITL logic.

• haystack_experimental/tools/types/protocol.py

  • Introduces ConfirmationPrompt and ExecutionPolicy protocols to allow users to easily customize how their HITL Tools should work.

How did you test it?

Provided an example script called hitl_example.py which shows how to use these new utility functions to convert existing tools into ones that ask for confirmation. I've also reproduced it here

# SPDX-FileCopyrightText: 2022-present deepset GmbH <[email protected]>
#
# SPDX-License-Identifier: Apache-2.0

from rich.console import Console

from haystack.dataclasses import ChatMessage
from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.tools import create_tool_from_function

from haystack_experimental.tools.human_in_the_loop import (
    confirmation_wrapper,
    SimpleInputPrompt,
    RichConsolePrompt,
)


def get_bank_balance(account_id: str) -> str:
    return f"Balance for account {account_id} is $1,234.56"


balance_tool = create_tool_from_function(
    function=get_bank_balance,
    name="get_bank_balance",
    description="Get the bank balance for a given account ID.",
)

#
# Example: Run Tool individually with different Prompts
#

# Use the console version
cons = Console()
console_tool = confirmation_wrapper(balance_tool, RichConsolePrompt(cons))
cons.print("\n[bold]Using console confirmation tool:[/bold]")
res = console_tool.invoke(account_id="123456")
cons.print(f"\n[bold green]Result:[/bold green] {res}")

# Use the simple input version
simple_tool = confirmation_wrapper(balance_tool, SimpleInputPrompt())
print("\nUsing simple input confirmation tool:")
res = simple_tool.invoke(account_id="123456")
print(f"\nResult: {res}")


#
# Example: Running with an Agent
#

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4.1"),
    tools=[console_tool],  # or simple_tool
    system_prompt="""
You are a helpful financial assistant. Use the provided tool to get bank balances when needed.
"""
)

result = agent.run([ChatMessage.from_user("What's the balance of account 56789?")])
last_message = result["last_message"]
cons.print(f"\n[bold green]Agent Result:[/bold green] {last_message.text}")

Notes for the reviewer

I'd appreciate some guidance on the best place to add this to Haystack. I'm unsure if this should become a part of the main library or if this would be better served in a cookbook or a tutorial.

The nature of Human in the Loop confirmations I believe can become quite custom which is why I found it difficult to create a generic pattern. I attempted to do this by creating the Protocols ConfirmationPrompt and ExecutionPolicy so users could easily customize how their Human in the Loop tools would work.

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.
• I documented my code
• I ran pre-commit hooks and fixed any issue

[coveralls]

Pull Request Test Coverage Report for Build 17762784085

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-5.0%) to 62.436%

---

Totals
Change from base Build 17750256045:	-5.0%
Covered Lines:	610
Relevant Lines:	977

---

💛 - Coveralls

[anakin87]

I like the overall idea and left a few comments.

About where to add this: maybe experimental + a notebook in cookbook (your example + explanations) might work while we get feedback and refine the design.

[anakin87]

this is done to avoid circular imports or for other reasons?

[sjrl]

yeah circular imports. I'll double check it's actually needed and leave a comment if it is

[julian-risch]

The idea makes sense to me on the higher level. As Haystack users will rarely use tools with HITL without an Agent, I still wonder if users would like it more if they could do the HITL settings in the Agent instead of each tool. As an alternative to your implementation.
One could argue that a user doesn't really want to change the tool but they want to change how the agent is allowed to use the tool.

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4.1"),
    tools=[balance_tool, some_other_tool],
    hitl=[HITLSettings(
        confirmation_strategy=ConsoleInput(),  
        confirmation_policy=AlwaysAsk(),      
    ),
    HITLSettings(
        confirmation_strategy=ConsoleInput(),  
        confirmation_policy=AskOnce(),      
    )],
    system_prompt="""...""",
)

or, assuming for a moment that an agent never needs multiples confirmation strategies:

agent = Agent(
    chat_generator=OpenAIChatGenerator(model="gpt-4.1"),
    tools=[balance_tool, some_other_tool, a_third_tool],
    confirmation_strategy=ConsoleInput(),  
    confirmation_policies=[AlwaysAsk, AskOnce, NeverAsk],
    system_prompt="""...""",
)

[julian-risch]

When I read policy, I first thought this is a choice between AlwaysAsk, AskOnce, NeverAsk. Here, the user is always asked.

[sjrl]

I gotcha, would a rename to AlwaysAskPolicy help make it more clear?

[sjrl]

The idea makes sense to me on the higher level. As Haystack users will rarely use tools with HITL without an Agent, I still wonder if users would like it more if they could do the HITL settings in the Agent instead of each tool. As an alternative to your implementation. One could argue that a user doesn't really want to change the tool but they want to change how the agent is allowed to use the tool.

@julian-risch I was just thinking about this as an alternative this morning. This would also solve the other issue I was having which is how could we also support serialization which isn't really easily doable with the approach I have currently. This would definitely solve that. I'll give a look over your idea!

[sjrl]

Closing in favor of #369