Merge pull request #19 from heroku/http_simple_tests

(depends on Http simple streamable http) E2E remote & local integration tests

Author: hillarysanders

.github/workflows/test.yml

@@ -0,0 +1,79 @@
+name: MCP Tests
+
+on:
+  push:
+    branches:
+  pull_request:
+
+jobs:
+  ###########################################################################
+  # 1 - Local integration tests (always run)
+  ###########################################################################
+  local:
+    runs-on: ubuntu-latest
+
+    # Dummy key lets the clients authenticate against the local servers.
+    env:
+      API_KEY: ci-test-key
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      # Optional: cache wheels to speed up numpy / scipy installs
+      - uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install -r requirements-dev.txt
+
+      - name: Run pytest (local transports)
+        run: pytest -q
+
+
+  ###########################################################################
+  # 2 - Remote smoke-test (only when secrets are set)
+  #
+  #    • Put MCP_SERVER_URL  → “https://<app>.herokuapp.com”
+  #    • Put API_KEY         → same key you set as Heroku config var
+  #
+  #    The fixture auto-skips the remote case if these are missing, so
+  #    the job is conditionally *created* only when both secrets exist.
+  ###########################################################################
+  remote:
+    if: ${{ secrets.MCP_SERVER_URL != '' && secrets.API_KEY != '' }}
+    runs-on: ubuntu-latest
+
+    env:
+      MCP_SERVER_URL: ${{ secrets.MCP_SERVER_URL }}
+      API_KEY:       ${{ secrets.API_KEY }}
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - uses: actions/cache@v4
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements*.txt') }}
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+
+      # We reuse the *same* test-suite; the fixture detects MCP_SERVER_URL
+      # and adds the “remote” parameter automatically.
+      - name: Run pytest (remote smoke)
+        run: pytest -q

README.md

@@ -93,17 +93,17 @@ First run:
 export API_KEY=$(heroku config:get API_KEY -a $APP_NAME)
 ```
 
-In the following commands, use either `example_clients/test_streamable_http.py` if you ran the streamable HTTP server above, or `example_clients/test_sse.py` if you're running the SSE server.
+In the following commands, use either `example_clients/streamable_http_client.py` if you ran the streamable HTTP server above, or `example_clients/sse_client.py` if you're running the SSE server.
 
 List tools:
 ```bash
-python example_clients/test_streamable_http.py mcp list_tools | jq
+python example_clients/streamable_http_client.py mcp list_tools | jq
 ```
 
 Example tool call request:
 *NOTE: this will intentionally NOT work if you have set `STDIO_MODE_ONLY` to `true`.*
 ```bash
-python example_clients/test_streamable_http.py mcp call_tool --args '{
+python example_clients/streamable_http_client.py mcp call_tool --args '{
   "name": "code_exec_python",
   "arguments": {
     "code": "import numpy as np; print(np.random.rand(50).tolist())",
@@ -118,12 +118,12 @@ There are two ways to easily test out your MCP server in STDIO mode:
 #### 1. Local STDIO - Example Python STDIO Client
 List tools:
 ```bash
-python example_clients/test_stdio.py mcp list_tools | jq
+python example_clients/stdio_client.py mcp list_tools | jq
 ```
 
 Example tool call request:
 ```bash
-python example_clients/test_stdio.py mcp call_tool --args '{
+python example_clients/stdio_client.py mcp call_tool --args '{
   "name": "code_exec_python",
   "arguments": {
     "code": "import numpy as np; print(np.random.rand(50).tolist())",
@@ -176,12 +176,12 @@ There are two ways to test out your remote MCP server in STDIO mode:
 #### 1. Remote STDIO - Example Python STDIO Client, Running On-Server
 To run against your deployed code, you can run the example client code on your deployed server inside a one-off dyno:
 ```bash
-heroku run --app $APP_NAME -- bash -c 'python -m example_clients.test_stdio mcp list_tools | jq'
+heroku run --app $APP_NAME -- bash -c 'python -m example_clients.stdio_client mcp list_tools | jq'
 ```
 or:
 ```bash
 heroku run --app $APP_NAME -- bash -c '
-python -m example_clients.test_stdio mcp call_tool --args '\''{
+python -m example_clients.stdio_client mcp call_tool --args '\''{
   "name": "code_exec_python",
   "arguments": {
     "code": "import numpy as np; print(np.random.rand(50).tolist())",

example_clients/test_sse.py example_clients/sse_client.pyexample_clients/test_sse.py renamed to example_clients/sse_client.py

@@ -0,0 +1,4 @@
+[pytest]
+addopts = -ra
+asyncio_mode = auto
+asyncio_default_fixture_loop_scope = function

example_clients/test_stdio.py example_clients/stdio_client.pyexample_clients/test_stdio.py renamed to example_clients/stdio_client.py

@@ -1,5 +1,8 @@
-mcp==1.9.2
+mcp[client,server]==1.9.2
 fastapi==0.115.12
 uvicorn==0.34.2
 python-dotenv==1.1.0
 mando==0.8.2
+# --- dev / CI only -------------------------------------------------
+pytest>=8.0
+pytest-asyncio>=0.23

example_clients/test_streamable_http.py example_clients/streamable_http_client.pyexample_clients/test_streamable_http.py renamed to example_clients/streamable_http_client.py

@@ -0,0 +1,42 @@
+# End-to-End Test Suite (`tests/`)
+
+You know what's better than unit tests? End to end integration tests.
+
+These pytest tests run **every MCP transport** this repo supports, both locally and (optionally) against a deployed Heroku app.
+
+| Context ID        | Transport exercised                                       | Needs running **web** dyno? |
+|-------------------|-----------------------------------------------------------|-----------------------------|
+| `http_local`      | Streamable HTTP on `localhost:8000/mcp/`                  | — |
+| `sse_local`       | SSE on `localhost:8000/mcp/sse`                           | — |
+| `stdio_local`     | STDIO (example client boots its own server)               | — |
+| `remote`          | Transport named in **`$REMOTE_SERVER_TRANSPORT_MODULE`** (`streamable_http_server` or `sse_server`) served by your web dyno | **Yes** |
+| `remote_stdio`    | STDIO via a **one-off Heroku dyno**                       | **No** – works even at `web=0` |
+
+*If the web dyno is asleep or scaled to `0`, the `remote` tests auto-skip.
+`remote_stdio` still runs, because it spins up its own one-off dyno.*
+
+To scale up the number of web dynos running in your app to 1, run:
+```bash
+heroku ps:scale web=1 -a "$APP_NAME"
+```
+
+---
+
+## 1 · Install dependencies
+
+```bash
+# inside an activated venv
+pip install -r requirements.txt
+```
+
+## 2 · Run local transports only
+```bash
+pytest tests -q
+```
+
+## 3 - Run local & remote transports
+```bash
+MCP_SERVER_URL=$(heroku info -s -a "$APP_NAME" | grep web_url | cut -d= -f2 | tr -d '\n') \
+API_KEY=$(heroku config:get API_KEY -a "$APP_NAME") \
+pytest tests -q
+```

pytest.ini

@@ -0,0 +1,54 @@
+"""
+Launch one of the example_clients in a fresh subprocess and return STDOUT.
+"""
+from __future__ import annotations
+import asyncio, os, sys, subprocess, textwrap
+from pathlib import Path
+from typing import Sequence, Mapping
+
+ROOT      = Path(__file__).resolve().parents[1]
+EXAMPLES  = ROOT / "example_clients"
+PYTHON_EXE = sys.executable
+
+
+async def call_client(
+    module_name: str,
+    cli_args: Sequence[str],
+    extra_env: Mapping[str, str] | None = None,
+) -> str:
+    env = os.environ.copy() | (extra_env or {})
+
+    # ----- NEW: run STDIO client inside a Heroku one-off dyno ----------
+    if module_name == "remote_stdio":
+        app = env.get("APP_NAME")
+        if not app:
+            raise RuntimeError("APP_NAME env-var required for remote_stdio context")
+        cmd = [
+            "heroku", "run", "--exit-code", "--app", app, "--",
+            "python", "-m", "example_clients.stdio_client", "mcp", *cli_args,
+        ]
+    # -------------------------------------------------------------------
+    else:
+        cmd = [
+            PYTHON_EXE,
+            "-m", f"example_clients.{module_name}",
+            "mcp", *cli_args,
+        ]
+
+    proc = await asyncio.create_subprocess_exec(
+        *cmd, cwd=ROOT, env=env,
+        stdout=subprocess.PIPE, stderr=subprocess.PIPE
+    )
+    out_b, err_b = await proc.communicate()
+    out, err = out_b.decode(), err_b.decode()
+    if proc.returncode:
+        raise RuntimeError(
+            textwrap.dedent(
+                f"""
+                Client {module_name} exited with {proc.returncode}
+                CMD   : {' '.join(cmd)}
+                STDERR:
+                {err}"""
+            )
+        )
+    return out