Improve search tool to extract resolved urls

[amrit110]

This pull request introduces significant improvements to the search tool infrastructure and URL handling in the agent evaluation framework, focusing on providing agents with real, fetchable URLs from Google Search. The changes include a new redirect resolution utility, a complete overhaul of the search tool to return actual URLs, dependency updates to support these features, and expanded test coverage. There are also minor improvements and fixes in other modules.

Search tool improvements:

• Introduced a new google_search function and revamped create_google_search_tool to use FunctionTool, enabling agents to receive search results with real, resolved URLs instead of redirect links. This supports a research workflow where agents can search, fetch, verify, and answer using actual web sources. (aieng/agent_evals/tools/search.py, aieng/agent_evals/tools/__init__.py, tests/aieng/agent_evals/tools/test_search.py) [1] [2] [3] [4] [5] [6] [7]

• Added a new _redirect.py module that provides asynchronous utilities for resolving known redirect URLs (such as those from Vertex AI Search) to their final destinations, with caching and retry logic. This is used by the updated search tool to ensure agents get usable URLs. (aieng/agent_evals/tools/_redirect.py)

Dependency and configuration updates:

• Updated pyproject.toml to add new dependencies required for HTTP requests, PDF handling, and HTML conversion, and incremented the package version. Also added a new CLI entry point for the knowledge agent. (pyproject.toml) [1] [2]

Testing enhancements:

• Expanded tests for the new search tool, including checks that ensure returned URLs are real and not redirects, and that the response structure is as expected. (tests/aieng/agent_evals/tools/test_search.py)

Other improvements and fixes:

• Improved the report generation agent to accept both LocalExperimentItem and DatasetItemClient types, handling both dict and class attribute access patterns. (aieng/agent_evals/report_generation/evaluation.py) [1] [2] [3] [4]
• Small typing fixes caught by stricter mypy rules. (implementations/aml_investigation/agent.py) [1] [2] [3]

[lotif]

Why do we have two pyproject.toml files?

[amrit110]

One for the project as a whole mostly used for development, and one for the package.

[lotif]

I see. I believe there is a way to do it with a single top level toml file but low priority, we can investigate later.