Skip to content

Commit 326aa24

Browse files
olaservoclaude
andcommitted
Assert on structured output in the smoke tests
The smoke tests connected to each server, listed its tools, and stopped there, so they could not catch a broken structured result. mcp-test-client now calls every tool that declares an outputSchema and checks the answer against it: the result must carry structuredContent, and a tool with an array-rooted schema must return a top-level JSON array while an object-rooted one must return an object. That second check is the point - a server advertising {"type":"array"} and returning {"result":[...]} passes a tools/list-only test and fails this one. A tool with no output schema is left alone, so this is not a new requirement on any example. The client negotiates with mode "auto": one server/discover probe, falling back to the 2025-11-25 initialize handshake. That keeps it working against every example as it stands today while also being able to test a server that speaks 2026-07-28. Tool calls reach the live NWS API, so an unreachable upstream surfaces as an error result. Those are reported as a skip rather than a failure; someone else's outage should not fail the build. mock-mcp-server moves to the low-level Server and advertises two tools, one array-rooted and one object-rooted, with the schemas written out literally. The array-rooted one is deliberate: a client that compiles every declared outputSchema when it connects will fail here if it assumes an output schema is always {"type":"object"}, which is no longer true as of protocol revision 2026-07-28. utils.sh no longer swallows build output. Builds ran with >/dev/null 2>&1 unconditionally, so a compile failure left no binary behind and the test reported a downstream spawn ENOENT instead of the actual error. That is what makes a broken build hard to diagnose from CI logs alone, and modelcontextprotocol#143 is the current example of it. run_build prints the compiler output when a build fails. check_dependency returns instead of exiting, so a missing toolchain fails its own test and still lets the suite finish and print its summary. Binaries are resolved through a helper that accounts for the .exe suffix, so the suite is runnable on Windows as well as in CI. CI moves from Node 18 to Node 20, which the 2.0 MCP SDK packages require. The test helpers move to those packages for the same reason; the examples themselves are untouched by this change. The Go examples and the Rust client are still uncovered. Each needs a change inside its own directory first - both clients abort when no .env file is present, so neither can be driven without credentials - so that coverage lands alongside those changes rather than here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent 7b7f81e commit 326aa24

8 files changed

Lines changed: 337 additions & 1110 deletions

File tree

.github/workflows/ci.yml

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,10 +14,11 @@ jobs:
1414
- name: Checkout repository
1515
uses: actions/checkout@v4
1616

17+
# The 2.0 MCP SDK packages require Node 20 or newer.
1718
- name: Set up Node.js
1819
uses: actions/setup-node@v4
1920
with:
20-
node-version: "18"
21+
node-version: "20"
2122

2223
- name: Set up Python
2324
uses: actions/setup-python@v5

tests/README.md

Lines changed: 23 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,26 @@
11
# MCP Quickstart Smoke Tests
22

3-
This directory contains smoke tests for the MCP quickstart examples. These tests verify that all example servers and clients can start and respond correctly, without calling external APIs.
3+
This directory contains smoke tests for the MCP quickstart examples. These tests verify that all example servers and clients can start and respond correctly.
44

55
## Overview
66

77
The smoke tests verify:
88

9-
- **Servers**: Each weather server (Python, TypeScript, Rust) can start and respond to MCP protocol requests
10-
- **Clients**: Each MCP client (Python, TypeScript) can connect to a mock server and list tools
9+
- **Servers**: Each weather server (Python, TypeScript, Rust, Go) can start, respond to MCP protocol requests, and honour the output schemas it advertises
10+
- **Clients**: Each MCP client (Python, TypeScript, Go, Rust) can connect to a mock server and list tools
11+
12+
The Ruby examples are not covered: the `mcp` gem cannot negotiate protocol revision `2026-07-28`.
13+
14+
## Structured output
15+
16+
Listing tools is not enough to catch a broken structured result, so each server test also **calls** every tool that declares an `outputSchema` and checks the answer:
17+
18+
- the result must carry `structuredContent`, and it must conform to the declared schema (the SDK validates this and throws on a mismatch);
19+
- a tool with an array-rooted schema must return a **top-level JSON array**, and one with an object-rooted schema must return an object.
20+
21+
The array case is the one worth guarding. A server that advertises `{"type": "array"}` and then answers `{"result": [...]}` passes a tools/list-only test and fails this one.
22+
23+
Tool calls reach the live NWS API. When it is unreachable the tools return an error result, which the test reports as a skip rather than a failure — someone else's outage should not fail the build.
1124

1225
## Running Tests
1326

@@ -23,17 +36,18 @@ The smoke tests verify:
2336
- **uv** (Python package manager)
2437
- **Rust** stable
2538
- **Cargo** (for Rust builds)
39+
- **Go** 1.25+
2640

2741
## How It Works
2842

2943
### Server Tests
3044

3145
Each server test:
3246

33-
1. Builds/prepares the server if needed
47+
1. Builds/prepares the server if needed (a failed build prints its compiler output rather than swallowing it)
3448
2. Uses `mcp-test-client.ts` to connect to the server via stdio
35-
3. Sends MCP initialize and `tools/list` requests
36-
4. Verifies the server responds with a valid tool list
49+
3. Negotiates a protocol era with `mode: "auto"` — one `server/discover` probe, falling back to the `2025-11-25` `initialize` handshake
50+
4. Lists tools, then calls each tool that declares an `outputSchema` and checks the structured result against it
3751
5. Reports pass/fail
3852

3953
### Client Tests
@@ -68,7 +82,9 @@ node tests/helpers/build/mcp-test-client.js python weather.py
6882

6983
### mock-mcp-server.ts
7084

71-
A minimal MCP server that verifies clients call the `tools/list` method and returns an empty tool list. Used to test clients without requiring a real weather server. Exits with an error if the client doesn't call `tools/list`.
85+
A minimal MCP server that verifies clients call the `tools/list` method. Used to test clients without requiring a real weather server. Exits with an error if the client doesn't call `tools/list`.
86+
87+
It advertises two tools whose output schemas cover both shapes a structured result can take: an object root, and an array root. The array-rooted one is deliberate — a client that compiles every declared `outputSchema` up front, as the Go and Rust quickstart clients do, will fail here if it assumes an output schema is always `{"type": "object"}`.
7288

7389
**Usage**:
7490

tests/helpers/mcp-test-client.ts

Lines changed: 89 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,112 @@
11
#!/usr/bin/env node
22
/**
33
* Minimal MCP Test Client for testing servers
4-
* Connects to a server, initializes, and lists tools
4+
*
5+
* Connects to a server, initializes, lists tools, and then checks the tools
6+
* actually honour what they advertise:
7+
*
8+
* - every tool that declares an `outputSchema` is called, and the result must
9+
* carry `structuredContent` (the SDK validates it against the schema for us
10+
* and throws on a mismatch);
11+
* - a tool whose `outputSchema` is array-rooted must answer with a top-level
12+
* JSON array, not an object wrapping one.
13+
*
14+
* The second check is the point of the exercise. Listing tools alone would not
15+
* notice a server that advertises `{"type": "array"}` and then returns
16+
* `{"result": [...]}`.
517
*/
618

7-
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
8-
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";
19+
import { Client } from "@modelcontextprotocol/client";
20+
import { StdioClientTransport } from "@modelcontextprotocol/client/stdio";
21+
22+
/** Arguments to call a tool with, chosen by matching its name. */
23+
const TOOL_ARGUMENTS: { match: RegExp; args: Record<string, unknown> }[] = [
24+
{ match: /alert/i, args: { state: "CA" } },
25+
{ match: /forecast/i, args: { latitude: 38.5816, longitude: -121.4944 } },
26+
];
27+
28+
function argumentsFor(name: string): Record<string, unknown> | undefined {
29+
return TOOL_ARGUMENTS.find(({ match }) => match.test(name))?.args;
30+
}
31+
32+
/** The root `type` of a JSON Schema, when it declares a single one. */
33+
function rootType(schema: unknown): string | undefined {
34+
if (typeof schema !== "object" || schema === null) return undefined;
35+
const type = (schema as { type?: unknown }).type;
36+
return typeof type === "string" ? type : undefined;
37+
}
938

1039
async function testServer(command: string, args: string[]) {
1140
console.error(`Testing server: ${command} ${args.join(" ")}`);
1241

13-
const transport = new StdioClientTransport({
14-
command,
15-
args,
16-
});
42+
const transport = new StdioClientTransport({ command, args });
1743

44+
// `auto` probes for 2026-07-28 and falls back to the 2025-11-25 handshake, so
45+
// this one helper tests servers of either era.
1846
const client = new Client(
19-
{
20-
name: "mcp-test-client",
21-
version: "1.0.0",
22-
},
23-
{
24-
capabilities: {},
25-
}
47+
{ name: "mcp-test-client", version: "1.0.0" },
48+
{ capabilities: {}, versionNegotiation: { mode: "auto" } },
2649
);
2750

2851
try {
29-
// Connect to server
3052
await client.connect(transport);
3153
console.error("✓ Connected to server");
3254

33-
// List tools
3455
const { tools } = await client.listTools();
3556
console.error(`✓ Listed ${tools.length} tools`);
3657

37-
// Success
58+
let checked = 0;
59+
for (const tool of tools) {
60+
if (!tool.outputSchema) continue;
61+
62+
const toolArgs = argumentsFor(tool.name);
63+
if (!toolArgs) {
64+
console.error(` - ${tool.name}: no known arguments, skipping call`);
65+
continue;
66+
}
67+
68+
// A throw here is the SDK rejecting the result against `outputSchema`.
69+
const result = await client.callTool({
70+
name: tool.name,
71+
arguments: toolArgs,
72+
});
73+
74+
// Upstream (api.weather.gov) being unreachable surfaces as an error
75+
// result, which is a legitimate answer and carries no structured data.
76+
// Skip rather than fail the build on someone else's outage.
77+
if (result.isError) {
78+
console.error(` - ${tool.name}: returned an error result, skipping`);
79+
continue;
80+
}
81+
82+
if (result.structuredContent === undefined) {
83+
throw new Error(
84+
`${tool.name} declares an output schema but returned no structuredContent`,
85+
);
86+
}
87+
88+
const expected = rootType(tool.outputSchema);
89+
const isArray = Array.isArray(result.structuredContent);
90+
91+
if (expected === "array" && !isArray) {
92+
throw new Error(
93+
`${tool.name} declares an array-rooted output schema but returned ` +
94+
`${JSON.stringify(result.structuredContent).slice(0, 80)}`,
95+
);
96+
}
97+
if (expected === "object" && isArray) {
98+
throw new Error(
99+
`${tool.name} declares an object-rooted output schema but returned an array`,
100+
);
101+
}
102+
103+
console.error(
104+
`✓ ${tool.name}: structuredContent is ${isArray ? "a top-level array" : "an object"}, matching its schema`,
105+
);
106+
checked += 1;
107+
}
108+
109+
console.error(`✓ Verified structured output for ${checked} tools`);
38110
console.error("✓ Server test passed");
39111
await client.close();
40112
process.exit(0);

tests/helpers/mock-mcp-server.ts

Lines changed: 98 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,120 @@
11
#!/usr/bin/env node
22
/**
33
* Mock MCP Server for testing clients
4-
* Verifies that clients call the tools/list method and returns an empty tool list
4+
*
5+
* Verifies that clients call `tools/list`, and advertises tools whose output
6+
* schemas cover both shapes a structured result can take: an object root, and
7+
* an array root (allowed as of protocol revision 2026-07-28).
8+
*
9+
* The array-rooted tool is deliberate. Clients that compile every declared
10+
* `outputSchema` up front — as the Go and Rust quickstart clients do — will
11+
* fail here if they assume an output schema is always `{"type": "object"}`.
12+
*
13+
* This uses the low-level `Server` rather than `McpServer` so the schemas are
14+
* written out literally, which is the point of a mock: what goes on the wire is
15+
* exactly what is in this file.
516
*/
617

7-
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
8-
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
9-
import { ListToolsRequestSchema } from "@modelcontextprotocol/sdk/types.js";
18+
import type { Tool } from "@modelcontextprotocol/server";
19+
import { Server } from "@modelcontextprotocol/server";
20+
import { serveStdio } from "@modelcontextprotocol/server/stdio";
1021

11-
const server = new McpServer(
22+
// Track whether tools/list was called
23+
let toolsListCalled = false;
24+
25+
const TOOLS: Tool[] = [
1226
{
13-
name: "mock-test-server",
14-
version: "1.0.0",
27+
name: "get_alerts",
28+
description: "Mock alerts, returning a top-level array",
29+
inputSchema: {
30+
type: "object",
31+
properties: { state: { type: "string" } },
32+
required: ["state"],
33+
},
34+
// Array root: legal as of 2026-07-28, rejected by older revisions.
35+
outputSchema: {
36+
type: "array",
37+
items: {
38+
type: "object",
39+
properties: { event: { type: "string" }, area: { type: "string" } },
40+
required: ["event", "area"],
41+
},
42+
},
1543
},
1644
{
17-
capabilities: {
18-
tools: {},
45+
name: "get_forecast",
46+
description: "Mock forecast, returning an object",
47+
inputSchema: {
48+
type: "object",
49+
properties: {
50+
latitude: { type: "number" },
51+
longitude: { type: "number" },
52+
},
53+
required: ["latitude", "longitude"],
1954
},
20-
}
21-
);
55+
outputSchema: {
56+
type: "object",
57+
properties: {
58+
latitude: { type: "number" },
59+
longitude: { type: "number" },
60+
summary: { type: "string" },
61+
},
62+
required: ["latitude", "longitude", "summary"],
63+
},
64+
},
65+
];
2266

23-
// Track whether tools/list was called
24-
let toolsListCalled = false;
67+
function buildServer(): Server {
68+
const server = new Server(
69+
{ name: "mock-test-server", version: "1.0.0" },
70+
{ capabilities: { tools: {} } },
71+
);
2572

26-
// Override the default tools/list handler to track calls
27-
server.server.setRequestHandler(ListToolsRequestSchema, async () => {
28-
toolsListCalled = true;
29-
return { tools: [] };
30-
});
73+
server.setRequestHandler("tools/list", async () => {
74+
toolsListCalled = true;
75+
return { tools: TOOLS };
76+
});
77+
78+
server.setRequestHandler("tools/call", async (request) => {
79+
const { name, arguments: args = {} } = request.params;
80+
81+
if (name === "get_alerts") {
82+
const state = String((args as { state?: unknown }).state ?? "??");
83+
const alerts = [{ event: "Mock Warning", area: state }];
84+
return {
85+
content: [{ type: "text", text: `1 alert for ${state}` }],
86+
structuredContent: alerts,
87+
};
88+
}
89+
90+
if (name === "get_forecast") {
91+
const { latitude = 0, longitude = 0 } = args as {
92+
latitude?: number;
93+
longitude?: number;
94+
};
95+
const forecast = { latitude, longitude, summary: "Mock conditions" };
96+
return {
97+
content: [{ type: "text", text: forecast.summary }],
98+
structuredContent: forecast,
99+
};
100+
}
31101

32-
async function main() {
33-
const transport = new StdioServerTransport();
34-
await server.connect(transport);
35-
console.error("Mock MCP Server running on stdio");
102+
return {
103+
content: [{ type: "text", text: `Unknown tool: ${name}` }],
104+
isError: true,
105+
};
106+
});
107+
108+
return server;
36109
}
37110

111+
serveStdio(buildServer);
112+
console.error("Mock MCP Server running on stdio");
113+
38114
// Verify that tools/list was called when the connection closes
39115
process.stdin.on("end", () => {
40116
if (!toolsListCalled) {
41117
console.error("Error: Client did not call tools/list");
42118
process.exit(1);
43119
}
44120
});
45-
46-
main().catch((error) => {
47-
console.error("Server error:", error);
48-
process.exit(1);
49-
});

0 commit comments

Comments
 (0)