Bug: tools declared by a custom_agents agent are announced but not callable by the model

[bancroftway]

Bug: tools declared by a custom_agents agent are announced but not callable by the model

Summary

When a session is created with custom_agents=[...] plus agent="<name>", the tools that agent
declares in its own tools: list are not usable by the model. The runtime reports them as
selected (a subagent.selected event lists them verbatim), but the model behaves as though it has
no tools at all: it makes zero tool calls and states plainly that it has no tool capable of
doing the work.

Passing the identical tool list via available_tools=[...] on the same model, same prompt,
same sandbox works correctly.

The failure is silent. There is no error, no warning, and no rejected tool call — the model simply
answers in prose. A caller that only inspects the final message (or a subagent.selected event)
sees a perfectly healthy-looking session.

Environment

Python SDK	github-copilot-sdk 1.0.11
CLI / server	copilot 1.0.79 (also reproduced on 1.0.78)
Connection	RuntimeConnection.for_uri(...) to a copilot --headless --server process in a container
Models	reproduced with gpt-5.3-codex and gpt-5.4
agent_mode	"autopilot"
Permissions	PermissionHandler.approve_all
OS	server on Linux (Ubuntu 24.04 container); client on Windows

Reproduction

repro.py below runs the same request twice against the same server: once with custom_agents,
once with available_tools. Start a headless server first, then run it.

# a copilot --headless --server on :3000, published to 18097, with a /workspace/repo cwd
docker run -d --rm --name ghcprepro -p 18097:3000 \
  -e COPILOT_SDK_AUTH_TOKEN="$GITHUB_TOKEN" \
  -e COPILOT_CONNECTION_TOKEN="repro123" \
  <image running: copilot --headless --no-auto-update --auth-token-env COPILOT_SDK_AUTH_TOKEN \
                          --no-auto-login --host 0.0.0.0 --port 3000>

python repro.py gpt-5.3-codex
docker exec ghcprepro sh -c 'ls /workspace/repo/'   # which file actually got created?

"""Minimal A/B: custom_agents' declared tools vs the identical list via available_tools."""
import asyncio, sys
from copilot import CopilotClient, RuntimeConnection
from copilot.session import PermissionHandler
from copilot.session_events import SessionEventType

URI, TOKEN = "localhost:18097", "repro123"
MODEL = sys.argv[1] if len(sys.argv) > 1 else "gpt-5.3-codex"
TOOLS = ["builtin:view", "builtin:grep", "builtin:glob",
         "builtin:edit", "builtin:create", "builtin:apply_patch", "builtin:skill"]

WRITER_AGENT = {
    "name": "writer",
    "description": "Creates files on disk.",
    "tools": TOOLS,                       # <-- identical list to available_tools in case B
    "model": MODEL,
    "prompt": "You are the Writer Agent. You create files on disk with your file tools.",
}

ASK = ("Create a file named {name} in the current directory containing exactly the text 'hello'. "
       "Then tell me in one sentence which tool you used, or state plainly that you have no tool "
       "capable of creating a file.")

async def run_case(name: str, *, use_custom_agent: bool) -> str:
    connection = RuntimeConnection.for_uri(URI, connection_token=TOKEN)
    async with CopilotClient(connection=connection, log_level="error") as client:
        kwargs = {"on_permission_request": PermissionHandler.approve_all, "model": MODEL,
                  "streaming": True, "working_directory": "/workspace/repo"}
        if use_custom_agent:
            kwargs["custom_agents"] = [WRITER_AGENT]
            kwargs["agent"] = "writer"
        else:
            kwargs["available_tools"] = TOOLS
        session = await client.create_session(**kwargs)

        done, text = asyncio.Event(), []
        def on_event(e):
            if e.type == SessionEventType.ASSISTANT_MESSAGE: text.append(e.data.content or "")
            elif e.type in (SessionEventType.SESSION_IDLE, SessionEventType.SESSION_ERROR): done.set()
        session.on(on_event)
        await session.send(ASK.format(name=name), agent_mode="autopilot")
        await asyncio.wait_for(done.wait(), timeout=120)
        return f"session={session.session_id} :: {' '.join(text).strip()[:200]}"

async def main():
    print("A custom_agents  ->", await run_case("case_a.txt", use_custom_agent=True))
    print("B available_tools->", await run_case("case_b.txt", use_custom_agent=False))

asyncio.run(main())

Actual result

A custom_agents  -> session=41113715-... :: I have no tool capable of creating a file.
B available_tools-> session=396f85a7-... :: Creating `case_b.txt` now ... I used the `apply_patch`
                                            tool to create `case_b.txt` with exactly `hello`.

$ ls /workspace/repo/
case_b.txt        # case_a.txt was never created

Case A reproduces on every attempt (3/3 trials, plus both models above).

The contradiction, from the server's own session log

~/.copilot/session-state/<session-id>/events.jsonl for the failing case A session announces
exactly the tools that the model then cannot use:

{"type":"subagent.selected","data":{"agentName":"writer","agentDisplayName":"writer",
 "tools":["builtin:view","builtin:grep","builtin:glob","builtin:edit","builtin:create",
          "builtin:apply_patch","builtin:skill"]}}

Tool invocations recorded in each session:

session	tool.execution_start entries
A (custom_agents)	none at all
B (available_tools)	apply_patch ×1

So the agent's tool set is resolved and reported, but never reaches the model's callable tool set.

Expected result

An agent's declared tools: should be callable by the model in that agent's turn — equivalent to
passing the same list via available_tools. Case A should create case_a.txt.

Failing that, a session whose agent declares tools the runtime cannot expose should raise an error
at create_session, rather than silently producing a tool-less agent.

Impact

This is expensive to diagnose because every observable signal says the session is healthy: the
agent is selected, the declared tools are echoed back, agent_mode is autopilot, permissions are
auto-approved, and the model returns a confident, well-formed answer. Only the filesystem (or the
absence of tool.execution_start events) reveals that nothing happened.

In our pipeline this manifested as agents that "completed" substantial work in ~18 seconds while
writing nothing to disk, and as automated fix-up steps that never repaired anything. Because each
affected stage then failed a downstream check for an unrelated-looking reason, it took a long time
to trace back to tool availability.

Workaround

Do not use custom_agents for any agent that needs tools. Pass the tool list via
available_tools on create_session, and supply the agent's instructions as a normal system
message instead of an agent definition.

Secondary observation (possibly intended, but surprising)

builtin:create alone is not sufficient for file creation — with
[view, grep, glob, edit, create] the model reports it has no way to create a file, and creation
only succeeds once builtin:apply_patch is included (the model then uses apply_patch). If
create is not independently usable, it would help for its absence/aliasing to be documented, or
for create to be rejected as unknown rather than accepted silently.

[github-actions]

Investigation findings

Label applied: bug — the behavior described is not working as intended.

What I investigated

• Python SDK — _convert_custom_agent_to_wire_format in python/copilot/client.py (line ~4026): the tools key is passed through directly and correctly into the wire payload under the agent definition.
• Node.js SDK — toWireCustomAgents in nodejs/src/client.ts (line 249): same conclusion — the function only rewrites mcpServers; all other fields including tools are spread into the wire object as-is.
• The top-level availableTools field and the per-agent tools field are different paths in the session payload. The SDK correctly sends both when the caller provides them.

What I found

The SDK is doing the right thing. The wire payload for the failing case includes:

{
  "customAgents": [
    { "name": "writer", "tools": ["builtin:view", "builtin:grep", ...], ... }
  ]
}

The CLI runtime acknowledges this by emitting a subagent.selected event that lists the declared tools — confirming they are received and resolved. However, the tools are never actually provided to the model's callable tool set, so the model can't use them and truthfully says it has no tool for the task.

The root cause is in the CLI runtime (server side), not in the SDK. When an agent is selected via customAgents, the runtime resolves and announces the agent's tools but does not wire them into the model's tool table for that turn. The top-level availableTools path does correctly reach the model, which is why the workaround works.

The failure is especially harmful because every observable signal (agent selection, tool list echoed, no errors, well-formed model response) looks healthy. Only the absence of tool.execution_start events or the lack of side effects reveals the problem.

Summary

This is a confirmed bug: custom agent tools declarations are processed and announced by the runtime but not made callable by the model. The SDK correctly serializes them. A fix is needed in the CLI runtime to ensure that an agent's declared tools are surfaced to the model in the same way availableTools is.

The secondary observation (that builtin:create alone appears insufficient for file creation) may be a separate, independent issue worth tracking.

Generated by Bug Handler for #2356 · sonnet46 33.4 AIC · ⌖ 5.24 AIC · ⊞ 5.9K · ◷