Skip to content

Conversation

@yuval-qf
Copy link
Collaborator

@yuval-qf yuval-qf commented Oct 19, 2025

Description

Motivation and Context

Type of Change

  • πŸ› Bug fix (non-breaking change which fixes an issue)
  • ✨ New feature (non-breaking change which adds functionality)
  • πŸ’₯ Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • πŸ“ Documentation update
  • 🎨 Code style/refactoring (no functional changes)
  • πŸ§ͺ Test updates
  • πŸ”§ Configuration/build changes

Changes Made

Screenshots/Examples (if applicable)

Checklist

  • I have read the CONTRIBUTING.md guide
  • My code follows the code style of this project (PEP 8, type hints, docstrings)
  • I have run uv run black . to format my code
  • I have run uv run flake8 . and fixed all issues
  • I have run uv run mypy --config-file .mypy.ini . and addressed type checking issues
  • I have run uv run bandit -c .bandit.yaml -r . for security checks
  • I have added tests that prove my fix is effective or that my feature works
  • I have run uv run pytest and all tests pass
  • I have manually tested my changes
  • I have updated the documentation accordingly
  • I have added/updated type hints for new/modified functions
  • My changes generate no new warnings
  • I have checked my code for security issues
  • Any dependent changes have been merged and published

Testing

Test Configuration:

  • Python version:
  • OS:
  • Other relevant details:

Test Steps:
1.
2.
3.

Additional Notes

Related Issues/PRs

  • Fixes #
  • Related to #
  • Depends on #

@yuval-qf yuval-qf self-assigned this Oct 19, 2025
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Oct 19, 2025

Summary by CodeRabbit

  • New Features

    • Added support for MCP (Multi-Component Protocol) communication alongside existing A2A protocol for agent evaluation
    • Introduced configurable transport options (HTTP, StreamableHTTP, SSE) for evaluation workflows
    • Enhanced CLI interface with transport selection for example applications
  • Chores

    • Added fastmcp dependency
    • Simplified test execution workflow
    • Updated code formatting tools configuration

Walkthrough

Threads Protocol and Transport through SDK, server, and evaluator-agent layers; introduces BaseEvaluatorAgent with A2A and MCP implementations plus a factory; updates orchestrator/runner to use the factory and async context managers; adds MCP example CLI and FastMCP dependency; simplifies CI/test invocation scripts.

Changes

Cohort / File(s) Summary
CI / Workflows
\.github/actions/run-tests/action.yml, \.github/workflows/rogue.yml, \.github/workflows/test.yml
Removed uv run wrappers, simplified test/agent invocation, consolidated install steps and added prune-cache: false in setup-uv.
Dependencies & Tooling
pyproject.toml, lefthook.yaml
Added fastmcp>=2.12.5; changed isort invocation to --profile black.
SDK types & surface
sdks/python/rogue_sdk/types.py, sdks/python/rogue_sdk/__init__.py, sdks/python/rogue_sdk/sdk.py
Added Protocol and Transport enums, AgentConfig fields protocol/transport, SecretStr auth support; exported Protocol/Transport; RogueSDK.run_evaluation gains judge_llm_api_key, protocol, transport params.
Evaluator package exports
rogue/evaluator_agent/__init__.py
Expanded public exports to include a2a, mcp, base_evaluator_agent, evaluator_agent_factory, A2AEvaluatorAgent, MCPEvaluatorAgent.
Base evaluator API
rogue/evaluator_agent/base_evaluator_agent.py
Replaced prior EvaluatorAgent with abstract BaseEvaluatorAgent; adds protocol/transport/headers handling, async context manager, chat-history helper, and abstract _send_message_to_evaluated_agent.
Factory & runner wiring
rogue/evaluator_agent/evaluator_agent_factory.py, rogue/evaluator_agent/run_evaluator_agent.py
Added get_evaluator_agent factory; arun_evaluator_agent/run_evaluator_agent accept and propagate protocol and transport, create agents via factory and use async with lifecycle.
A2A evaluator implementation
rogue/evaluator_agent/a2a/*
Added A2AEvaluatorAgent with lazy evaluated-agent client resolution, message send/parse flow, chat-history updates, async lifecycle, and detailed logging.
MCP evaluator implementation
rogue/evaluator_agent/mcp/*
Added MCPEvaluatorAgent supporting Streamable-HTTP and SSE transports, MCP client/tool invocation, response aggregation, chat-history updates, and error handling.
Server wiring
rogue/server/core/evaluation_orchestrator.py, rogue/server/services/*.py
Threaded protocol and transport through EvaluationOrchestrator, ScenarioEvaluationService, EvaluationService, and evaluation library/service calls so evaluator runner receives protocol/transport.
Examples & wrappers
examples/mcp/tshirt_store_langgraph_mcp/*, examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py, rogue/__main__.py, .vscode/launch.json
Added MCP example CLI with transport option, memoized get_mcp_server, switched to relative imports, session_id extraction in tool, registered new example in CLI, updated VSCode launch entries.

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant SDK as RogueSDK
    participant Orch as EvaluationOrchestrator
    participant Factory as get_evaluator_agent
    participant Agent as BaseEvaluatorAgent
    participant A2A as A2AEvaluatorAgent
    participant MCP as MCPEvaluatorAgent
    participant Eval as EvaluatedAgent

    SDK->>Orch: run_job(protocol, transport, ...)
    Orch->>Factory: get_evaluator_agent(protocol, transport, ...)
    alt protocol == A2A
        Factory-->>A2A: instantiate(...)
        Orch->>A2A: async with A2A
        Orch->>A2A: _send_message_to_evaluated_agent(ctx,msg)
        A2A->>Eval: A2A HTTP request
        Eval-->>A2A: response
        A2A-->>Orch: response dict
    else protocol == MCP
        Factory-->>MCP: instantiate(...)
        Orch->>MCP: async with MCP
        Orch->>MCP: _send_message_to_evaluated_agent(ctx,msg)
        MCP->>Eval: MCP tool invocation (stream/request)
        Eval-->>MCP: streamed/aggregated response
        MCP-->>Orch: response dict
    end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Suggested reviewers

  • drorIvry
  • osher-qualifire

Poem

πŸ‡ I nibbled enums and threaded transports bright,
I hopped from base to A2A and MCP light,
A factory bloom, async doors swing wide,
Examples warmed, CI trimmed the tide,
β€” a joyful rabbit, thumping with pride.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Description Check ⚠️ Warning The pull request description is substantially incomplete despite all checklist items being marked as complete. The required sections from the templateβ€”Description, Motivation and Context, Changes Made, Testing, and Related Issues/PRsβ€”contain only placeholder comments or are entirely empty. While the author checked "New feature" under Type of Change and marked all quality/process checklist items as done, they have not filled in the core descriptive content that explains what changes were made, why they were necessary, how to test them, or which issues this addresses. This leaves reviewers without essential context despite the extensive changes shown in the raw summary. The author should complete all empty sections in the description template. Specifically: add a detailed description of the MCP transport changes; explain the motivation (e.g., why MCP transport support was needed); list the specific changes made (referenced in the raw summary but not documented in the PR); provide actual test configuration details and reproduction steps; and link the FIRE-831 ticket and any other related issues under "Related Issues/PRs." The completed checklist indicates this work was done, so documenting it is necessary for reviewers to understand the scope and rationale.
βœ… Passed checks (2 passed)
Check name Status Explanation
Title Check βœ… Passed The title "FIRE-831 | Rogue Server | MCP transport" clearly describes the primary change evident in the raw summary: adding MCP (Multi-Component Protocol) transport support to the Rogue Server. The changeset includes new Protocol and Transport enums, MCPEvaluatorAgent and A2AEvaluatorAgent classes, factory-based dispatching, and MCP integration examples. The title is specific and directly references the main feature addition without extraneous noise, allowing a developer scanning history to understand this PR introduces MCP transport functionality.
Docstring Coverage βœ… Passed No functions found in the changes. Docstring coverage check skipped.
✨ Finishing touches
πŸ§ͺ Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch feature/FIRE-831-mcp-transport-support

πŸ“œ Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 24f1e74 and b23a8f1.

πŸ“’ Files selected for processing (1)
  • .github/workflows/rogue.yml (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • .github/workflows/rogue.yml
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: codestyle
  • GitHub Check: rogue_sanity

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

♻️ Duplicate comments (2)
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1)

26-27: Use the documented import path for fastmcp transports.

The current import uses from fastmcp.client import ..., but the official fastmcp documentation (for versions 2.12.5+) consistently shows transports should be imported from fastmcp.client.transports. While the current path may work, using the documented path improves maintainability and aligns with fastmcp conventions.

Apply this diff:

         from fastmcp import Client
-        from fastmcp.client import SSETransport, StreamableHttpTransport
+        from fastmcp.client.transports import SSETransport, StreamableHttpTransport

Based on learnings.

rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (1)

187-187: Response format inconsistency between A2A and MCP protocols.

Line 187 returns the full JSON payload (response.model_dump_json()), while the extracted text (agent_response_text) is stored in chat history. The MCP implementation returns extracted text in the "response" key. This protocol-level inconsistency can degrade LLM tool reasoning and complicate downstream processing.

Based on past review comments, consider normalizing to return extracted text:

-            return {"response": response.model_dump_json()}
+            return {
+                "response": agent_response_text,
+                "raw": response.model_dump_json(),  # optional: keep full payload
+            }
🧹 Nitpick comments (2)
rogue/evaluator_agent/evaluator_agent_factory.py (1)

10-23: Consider adding a docstring to document the factory function.

The function signature is well-typed, but a docstring would help document the factory's purpose, parameter meanings, and return behavior for maintainers.

Apply this diff:

 def get_evaluator_agent(
     protocol: Protocol,
     transport: Transport | None,
     evaluated_agent_address: str,
     judge_llm: str,
     scenarios: Scenarios,
     business_context: Optional[str],
     headers: Optional[dict[str, str]] = None,
     judge_llm_auth: Optional[str] = None,
     debug: bool = False,
     deep_test_mode: bool = False,
     chat_update_callback: Optional[Callable[[dict], None]] = None,
     **kwargs,
 ) -> BaseEvaluatorAgent:
+    """
+    Factory function to create protocol-specific evaluator agents.
+    
+    :param protocol: Protocol type (A2A or MCP)
+    :param transport: Transport type (or None to use protocol default)
+    :param evaluated_agent_address: URL of the agent to evaluate
+    :param judge_llm: Model name for the judge LLM
+    :param scenarios: Collection of evaluation scenarios
+    :param business_context: Business context for evaluation
+    :param headers: Optional HTTP headers
+    :param judge_llm_auth: Optional API key for judge LLM
+    :param debug: Enable debug logging
+    :param deep_test_mode: Enable deep testing mode
+    :param chat_update_callback: Optional callback for chat updates
+    :return: Protocol-specific evaluator agent instance
+    :raises ValueError: If protocol is not supported
+    """
     if protocol == Protocol.A2A:
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1)

131-144: Add error_type to exception logging for consistency with A2A observability.

The exception handler logs structured context but omits error_type, which the A2A implementation includes for debugging. Adding it improves observability parity across protocols.

Apply this diff:

         except Exception as e:
             logger.exception(
                 "Error while sending message to evaluated agent using mcp",
                 extra={
                     "protocol": "mcp",
                     "agent_url": self._evaluated_agent_address,
                     "transport": self._transport.value,
                     "message": message[:100] + "..." if len(message) > 100 else message,
+                    "error": str(e),
+                    "error_type": type(e).__name__,
                 },
             )
πŸ“œ Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between dfbb5ff and dd87e37.

πŸ“’ Files selected for processing (5)
  • rogue/evaluator_agent/evaluator_agent_factory.py (1 hunks)
  • rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (1 hunks)
  • rogue/evaluator_agent/protocols/base_evaluator_agent.py (7 hunks)
  • rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1 hunks)
  • sdks/python/rogue_sdk/types.py (4 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • sdks/python/rogue_sdk/types.py
🧰 Additional context used
πŸ““ Path-based instructions (1)
**/*.py

πŸ“„ CodeRabbit inference engine (AGENTS.md)

**/*.py: Format Python code with Black
Ensure code passes flake8 linting
Run mypy with the repository configuration for static typing
Run Bandit security checks using .bandit.yaml configuration
Use isort import conventions for import ordering
Add type hints to all function signatures
Follow PEP 8 naming (snake_case for variables/functions, PascalCase for classes)
Use try/except around code that may raise exceptions

Files:

  • rogue/evaluator_agent/protocols/a2a_evaluator_agent.py
  • rogue/evaluator_agent/evaluator_agent_factory.py
  • rogue/evaluator_agent/protocols/mcp_evaluator_agent.py
  • rogue/evaluator_agent/protocols/base_evaluator_agent.py
🧬 Code graph analysis (4)
rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (4)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/common/remote_agent_connection.py (1)
  • RemoteAgentConnections (52-130)
rogue/evaluator_agent/protocols/base_evaluator_agent.py (3)
  • BaseEvaluatorAgent (156-556)
  • _send_message_to_evaluated_agent (491-508)
  • _add_message_to_chat_history (519-545)
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (75-108)
rogue/evaluator_agent/evaluator_agent_factory.py (4)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (1)
  • A2AEvaluatorAgent (19-212)
rogue/evaluator_agent/protocols/base_evaluator_agent.py (1)
  • BaseEvaluatorAgent (156-556)
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1)
  • MCPEvaluatorAgent (10-144)
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (3)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/evaluator_agent/protocols/base_evaluator_agent.py (3)
  • BaseEvaluatorAgent (156-556)
  • _send_message_to_evaluated_agent (491-508)
  • _add_message_to_chat_history (519-545)
rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (113-199)
rogue/evaluator_agent/protocols/base_evaluator_agent.py (5)
sdks/python/rogue_sdk/types.py (11)
  • Protocol (68-79)
  • Scenario (139-178)
  • Scenarios (181-199)
  • ScenarioType (51-55)
  • Transport (82-93)
  • get_default_transport (74-79)
  • is_valid_for_protocol (92-93)
  • EvaluationResults (237-253)
  • ChatHistory (210-218)
  • add_message (215-218)
  • ChatMessage (202-207)
rogue/common/agent_model_wrapper.py (1)
  • get_llm_from_model (11-35)
rogue/evaluator_agent/policy_evaluation.py (1)
  • evaluate_policy (136-204)
rogue/evaluator_agent/protocols/a2a_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (113-199)
rogue/evaluator_agent/protocols/mcp_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (75-108)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: rogue_sanity
  • GitHub Check: codestyle

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)

176-189: Normalize return shape: put text in response and include raw payload.

Aligns with MCP and avoids forcing callers to parse JSON-strings.

-            return {"response": response.model_dump_json()}
+            return {
+                "response": agent_response_text,
+                "raw": response.model_dump(),  # keep structured payload if needed
+            }
🧹 Nitpick comments (4)
rogue/evaluator_agent/mcp/__init__.py (1)

1-2: Drop redundant submodule import to reduce import-time side effects.

Importing the module then the class is unnecessary and can contribute to cycles.

-from . import mcp_evaluator_agent
 from .mcp_evaluator_agent import MCPEvaluatorAgent
rogue/evaluator_agent/a2a/__init__.py (1)

1-2: Remove unnecessary submodule import.

Keeps the surface the same while minimizing circular-import risk.

-from . import a2a_evaluator_agent
 from .a2a_evaluator_agent import A2AEvaluatorAgent
rogue/evaluator_agent/__init__.py (1)

1-11: Consider slimming package imports to reduce future cycle risk.

Re-exporting classes is fine; importing heavy modules (e.g., evaluator_agent_factory) at package import time isn’t necessary. Consider dropping those from the top-level import list and all, keeping only subpackages and class re-exports.

Also applies to: 12-21

rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)

132-141: Enrich MCP exception logs with error details to match A2A observability.

Add error and error_type fields so logs are consistent and actionable.

-            logger.exception(
-                "Error while sending message to evaluated agent using mcp",
+            logger.exception(
+                "❌ MCP call failed - error sending message to evaluated agent",
                 extra={
                     "protocol": "mcp",
                     "agent_url": self._evaluated_agent_address,
                     "transport": self._transport.value,
                     "message": message[:100] + "..." if len(message) > 100 else message,
+                    "error": str(e),
+                    "error_type": type(e).__name__,
                 },
             )
πŸ“œ Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between dd87e37 and e67b99e.

πŸ“’ Files selected for processing (7)
  • rogue/evaluator_agent/__init__.py (1 hunks)
  • rogue/evaluator_agent/a2a/__init__.py (1 hunks)
  • rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1 hunks)
  • rogue/evaluator_agent/base_evaluator_agent.py (7 hunks)
  • rogue/evaluator_agent/evaluator_agent_factory.py (1 hunks)
  • rogue/evaluator_agent/mcp/__init__.py (1 hunks)
  • rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1 hunks)
🧰 Additional context used
πŸ““ Path-based instructions (1)
**/*.py

πŸ“„ CodeRabbit inference engine (AGENTS.md)

**/*.py: Format Python code with Black
Ensure code passes flake8 linting
Run mypy with the repository configuration for static typing
Run Bandit security checks using .bandit.yaml configuration
Use isort import conventions for import ordering
Add type hints to all function signatures
Follow PEP 8 naming (snake_case for variables/functions, PascalCase for classes)
Use try/except around code that may raise exceptions

Files:

  • rogue/evaluator_agent/mcp/__init__.py
  • rogue/evaluator_agent/mcp/mcp_evaluator_agent.py
  • rogue/evaluator_agent/a2a/__init__.py
  • rogue/evaluator_agent/base_evaluator_agent.py
  • rogue/evaluator_agent/__init__.py
  • rogue/evaluator_agent/evaluator_agent_factory.py
  • rogue/evaluator_agent/a2a/a2a_evaluator_agent.py
🧬 Code graph analysis (7)
rogue/evaluator_agent/mcp/__init__.py (1)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)
  • MCPEvaluatorAgent (10-145)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (3)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/evaluator_agent/base_evaluator_agent.py (3)
  • BaseEvaluatorAgent (156-556)
  • _send_message_to_evaluated_agent (491-508)
  • _add_message_to_chat_history (519-545)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (114-200)
rogue/evaluator_agent/a2a/__init__.py (1)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)
  • A2AEvaluatorAgent (19-213)
rogue/evaluator_agent/base_evaluator_agent.py (5)
sdks/python/rogue_sdk/types.py (7)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
  • get_default_transport (74-79)
  • is_valid_for_protocol (92-93)
  • ChatHistory (210-218)
  • add_message (215-218)
rogue/common/agent_model_wrapper.py (1)
  • get_llm_from_model (11-35)
rogue/evaluator_agent/policy_evaluation.py (1)
  • evaluate_policy (136-204)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (114-200)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (75-109)
rogue/evaluator_agent/__init__.py (3)
rogue/evaluator_agent/run_evaluator_agent.py (1)
  • run_evaluator_agent (220-248)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)
  • A2AEvaluatorAgent (19-213)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)
  • MCPEvaluatorAgent (10-145)
rogue/evaluator_agent/evaluator_agent_factory.py (4)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (1)
  • A2AEvaluatorAgent (19-213)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)
  • MCPEvaluatorAgent (10-145)
rogue/evaluator_agent/base_evaluator_agent.py (1)
  • BaseEvaluatorAgent (156-556)
rogue/evaluator_agent/a2a/a2a_evaluator_agent.py (4)
sdks/python/rogue_sdk/types.py (3)
  • Protocol (68-79)
  • Scenarios (181-199)
  • Transport (82-93)
rogue/common/remote_agent_connection.py (1)
  • RemoteAgentConnections (52-130)
rogue/evaluator_agent/base_evaluator_agent.py (3)
  • BaseEvaluatorAgent (156-556)
  • _send_message_to_evaluated_agent (491-508)
  • _add_message_to_chat_history (519-545)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)
  • _send_message_to_evaluated_agent (75-109)
πŸͺ› GitHub Actions: Rogue
rogue/evaluator_agent/evaluator_agent_factory.py

[error] 1-1: ImportError: cannot import name 'A2AEvaluatorAgent' from partially initialized module 'rogue.evaluator_agent' during 'python -m rogue cli'. This is likely due to a circular import.

πŸͺ› GitHub Actions: Test
rogue/evaluator_agent/evaluator_agent_factory.py

[error] 1-1: ImportError: cannot import name 'A2AEvaluatorAgent' from partially initialized module 'rogue.evaluator_agent' (likely due to a circular import) during pytest collection.

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: codestyle
πŸ”‡ Additional comments (6)
rogue/evaluator_agent/mcp/mcp_evaluator_agent.py (1)

26-28: I need to verify the codebase's fastmcp version constraint and understand the actual compatibility concern.

Let me search the codebase for the fastmcp version constraint:

The review comment is based on an incorrect premise. The import paths don't vary across 2.x versions.

All current FastMCP documentation consistently shows imports from fastmcp.client.transports, not fastmcp.client. The official docs demonstrate from fastmcp.client.transports import StreamableHttpTransport across all 2.x versions shown (2.0.0 through current).

The code's current import path (from fastmcp.client import SSETransport, StreamableHttpTransport) is not documented anywhere. Rather than a backward compatibility issue with varying module layouts, the simple fix is to align with the documented standard import path. A try/except fallback is unnecessary and obscures the straightforward update needed.

Rewritten suggestion: Update the import to use the documented path: from fastmcp.client.transports import SSETransport, StreamableHttpTransport (no fallback needed).

Likely an incorrect or invalid review comment.

rogue/evaluator_agent/base_evaluator_agent.py (5)

156-156: LGTM!

The class name change to BaseEvaluatorAgent and inheriting from ABC clearly signals this is now an abstract base class, aligning well with the protocol/transport abstraction pattern.


157-189: LGTM!

The updated __init__ signature correctly introduces protocol/transport abstraction with:

  • Proper default transport resolution from protocol
  • Validation that transport is compatible with protocol
  • Support for protocol-specific headers
  • Extensibility via *args, **kwargs for subclasses

490-508: LGTM!

Correctly made abstract with @abstractmethod decorator and clear documentation. The relevant code snippets confirm that both A2AEvaluatorAgent (lines 113-199) and MCPEvaluatorAgent (lines 74-108) properly implement this method with protocol-specific logic.


519-546: LGTM!

Excellent refactor! This helper centralizes chat history management and callback synchronization, eliminating duplication between protocol-specific implementations. The role mapping logic for callbacks is clear and appropriate.


547-556: Base async context manager implementation is correct; subclasses already implement proper cleanup.

Both MCPEvaluatorAgent and A2AEvaluatorAgent correctly override __aexit__ to clean up their managed resources (MCP client and HTTP client respectively). The base BaseEvaluatorAgent implementation with no cleanup is appropriate since the base class doesn't manage resources. No changes needed.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
rogue/__main__.py (1)

131-132: Remove duplicate comment.

The comment "Use subprocess to run the example agent" is duplicated from lines 120-121. Consider removing it or consolidating the logic to reduce duplication.

Apply this diff:

 elif example_name == "tshirt_store_langgraph_mcp":
-    # Use subprocess to run the example agent
     cmd = [
πŸ“œ Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between c36e371 and 473e0bb.

πŸ“’ Files selected for processing (5)
  • .vscode/launch.json (1 hunks)
  • examples/mcp/tshirt_store_langgraph_mcp/__main__.py (1 hunks)
  • examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1 hunks)
  • rogue/__main__.py (2 hunks)
  • rogue/evaluator_agent/evaluator_agent_factory.py (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • rogue/evaluator_agent/evaluator_agent_factory.py
🧰 Additional context used
πŸ““ Path-based instructions (1)
**/*.py

πŸ“„ CodeRabbit inference engine (AGENTS.md)

**/*.py: Format Python code with Black
Ensure code passes flake8 linting
Run mypy with the repository configuration for static typing
Run Bandit security checks using .bandit.yaml configuration
Use isort import conventions for import ordering
Add type hints to all function signatures
Follow PEP 8 naming (snake_case for variables/functions, PascalCase for classes)
Use try/except around code that may raise exceptions

Files:

  • examples/mcp/tshirt_store_langgraph_mcp/__main__.py
  • examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py
  • rogue/__main__.py
🧬 Code graph analysis (1)
examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1)
examples/mcp/tshirt_store_langgraph_mcp/shirtify_agent.py (1)
  • ShirtifyAgent (64-136)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
  • GitHub Check: codestyle
πŸ”‡ Additional comments (4)
.vscode/launch.json (1)

61-61: Module paths are valid; .env files are local configuration (gitignored by design).

Both module paths (examples.tshirt_store_agent and examples.mcp.tshirt_store_langgraph_mcp) correctly reference importable Python modules with proper structure (init.py and main.py present). The envFile paths reference local configuration files that are explicitly excluded from version control (per .gitignore policy: "do not commit any .env files to git"). These files are expected to be created locally by developers and do not need to exist in the repository.

Likely an incorrect or invalid review comment.

examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1)

13-14: LGTM! Correct use of relative import.

The change from absolute to relative import is appropriate for a package-internal module reference. The import ordering follows isort conventions (third-party imports, then relative imports with a blank line separator).

examples/mcp/tshirt_store_langgraph_mcp/__main__.py (1)

2-3: LGTM! Correct import structure.

The relative import and blank line separator are appropriate. This follows isort conventions by placing a blank line between third-party imports (line 1) and package-relative imports (line 3).

rogue/__main__.py (1)

47-49: LGTM!

The parser extension properly adds the new MCP example choice and updates the help text accordingly.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (1)
examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1)

29-41: Refine exception handling to avoid masking unexpected errors.

The bare except Exception: on line 39 catches all exceptions during session extraction, which may mask unexpected errors like attribute errors or bugs in the extraction logic.

Consider catching more specific exceptions related to request context access, or at minimum, re-raise unexpected exceptions after logging.

Apply this diff to improve exception handling:

         session_id: str | None = None
         try:
             request: Request = context.request_context.request  # type: ignore
 
             # The session id should be in the headers for streamable-http transport
             session_id = request.headers.get("mcp-session-id")
 
             # The session id might also be in query param when using sse transport
             if session_id is None:
                 session_id = request.query_params.get("session_id")
-        except Exception:
+        except (AttributeError, KeyError, TypeError) as e:
             session_id = None
-            logger.exception("Error while extracting session id")
+            logger.exception(f"Error while extracting session id: {e}")
πŸ“œ Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 473e0bb and f28b81b.

πŸ“’ Files selected for processing (3)
  • examples/mcp/tshirt_store_langgraph_mcp/__main__.py (1 hunks)
  • examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1 hunks)
  • rogue/__main__.py (2 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • rogue/main.py
🧰 Additional context used
πŸ““ Path-based instructions (1)
**/*.py

πŸ“„ CodeRabbit inference engine (AGENTS.md)

**/*.py: Format Python code with Black
Ensure code passes flake8 linting
Run mypy with the repository configuration for static typing
Run Bandit security checks using .bandit.yaml configuration
Use isort import conventions for import ordering
Add type hints to all function signatures
Follow PEP 8 naming (snake_case for variables/functions, PascalCase for classes)
Use try/except around code that may raise exceptions

Files:

  • examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py
  • examples/mcp/tshirt_store_langgraph_mcp/__main__.py
🧬 Code graph analysis (2)
examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1)
examples/mcp/tshirt_store_langgraph_mcp/shirtify_agent.py (2)
  • ShirtifyAgent (64-136)
  • invoke (76-82)
examples/mcp/tshirt_store_langgraph_mcp/__main__.py (1)
examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (1)
  • get_mcp_server (19-50)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
  • GitHub Check: rogue_sanity
  • GitHub Check: codestyle
πŸ”‡ Additional comments (6)
examples/mcp/tshirt_store_langgraph_mcp/mcp_agent_wrapper.py (3)

9-10: LGTM!

The lru_cache import is appropriate for memoizing the MCP server factory function.


15-16: LGTM!

The relative import is appropriate for the package structure and consistent with the CLI module.


18-25: Critical: LRU cache will ignore host/port parameters after first call.

The @lru_cache(maxsize=1) decorator will cache only the first return value. Subsequent calls with different host or port values will return the cached instance created with the first call's parameters, silently ignoring the new values.

If the intent is to ensure a singleton MCP server, the parameters should be removed or validated. Otherwise, remove the cache or increase maxsize and ensure parameter-based caching works as expected.

Apply this diff to fix the issue by removing parameters (singleton approach):

-@lru_cache(maxsize=1)
-def get_mcp_server(host: str = "127.0.0.1", port: int = 10001) -> FastMCP:
+def get_mcp_server() -> FastMCP:
+    host = "127.0.0.1"
+    port = 10001
     agent = ShirtifyAgent()
     mcp = FastMCP(

Or remove caching to respect parameters:

-@lru_cache(maxsize=1)
 def get_mcp_server(host: str = "127.0.0.1", port: int = 10001) -> FastMCP:

Likely an incorrect or invalid review comment.

examples/mcp/tshirt_store_langgraph_mcp/__main__.py (3)

1-6: LGTM!

The imports are well-organized and appropriate. The relative import is consistent with the package structure, and the Literal type hint enhances type safety for the transport parameter.


11-21: LGTM!

The Click CLI setup is well-structured with appropriate defaults and type hints. The Literal type annotation for transport choices provides strong type safety.


23-28: Verify impact of the LRU cache issue in get_mcp_server.

The implementation correctly passes the CLI-provided host and port to get_mcp_server. However, due to the @lru_cache(maxsize=1) decorator in mcp_agent_wrapper.py, these parameters will be ignored after the first call, which undermines the CLI's flexibility.

Ensure the caching issue identified in mcp_agent_wrapper.py (lines 18-25) is resolved to make the CLI parameters effective.

@yuval-qf yuval-qf temporarily deployed to rogue-sanity-ci-secrets October 22, 2025 10:38 — with GitHub Actions Inactive
@yuval-qf yuval-qf temporarily deployed to rogue-sanity-ci-secrets October 22, 2025 10:46 — with GitHub Actions Inactive
@yuval-qf yuval-qf temporarily deployed to rogue-sanity-ci-secrets October 22, 2025 11:06 — with GitHub Actions Inactive
@yuval-qf yuval-qf temporarily deployed to rogue-sanity-ci-secrets October 22, 2025 11:15 — with GitHub Actions Inactive
@yuval-qf yuval-qf had a problem deploying to rogue-sanity-ci-secrets October 22, 2025 13:37 — with GitHub Actions Failure
@yuval-qf yuval-qf merged commit 2ec51dc into main Oct 22, 2025
8 of 9 checks passed
@yuval-qf yuval-qf deleted the feature/FIRE-831-mcp-transport-support branch October 22, 2025 13:40
@drorIvry drorIvry mentioned this pull request Oct 28, 2025
21 tasks
@drorIvry drorIvry mentioned this pull request Nov 10, 2025
21 tasks
@coderabbitai coderabbitai bot mentioned this pull request Jan 4, 2026
21 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants