cekura-benchmark

A Pipecat AI voice agent built with a cascade pipeline (STT → LLM → TTS).

One codebase serves two benchmark agents. The AGENT_DIR env var selects which agent definition a given deployment runs:

• AGENT_DIR=appointments → server/appointments/ (clinic scheduling agent)
• AGENT_DIR=insurance → server/insurance/ (Medicare insurance agent)

Each agent dir holds the four canonical definition files — system-prompt.txt, first-message.txt, tool-definitions.json (LLM tool schemas, used byte-exact), and mock-tools.json (canned tool results, resolved in-process). The bot fails at boot if AGENT_DIR is unset or invalid — a benchmark deployment never falls back to the wrong agent.

In production this means two Pipecat Cloud agents deployed from the identical image (cekura-benchmark-appointments and cekura-benchmark-insurance), differing only in that one env var, with two Twilio phone numbers — each number pointed at its own agent. See Deploying to Pipecat Cloud.

Configuration

• Bot Type: Telephony
• Transport(s): Twilio
• Pipeline: Cascade
  • STT: Deepgram
  • LLM: OpenAI Responses
  • TTS: Cartesia
• Features:
  • Krisp Noise Cancellation

Setup

Setting Up Twilio

1. Create a TwiML Bin

A TwiML Bin tells Twilio how to handle incoming calls. You'll create one that establishes a WebSocket connection to your bot.

1. Go to the Twilio Console

2. Navigate to TwiML Bins → My TwiML Bins

3. Click the + to create a new TwiML Bin

4. Name your bin and add the TwiML:

   For Local Development:

   <?xml version="1.0" encoding="UTF-8"?>
   <Response>
     <Connect>
       <Stream url="wss://your-url.ngrok.io/ws" />
     </Connect>
   </Response>

   Replace your-url.ngrok.io with your ngrok URL.

   For Pipecat Cloud:

   <?xml version="1.0" encoding="UTF-8"?>
   <Response>
     <Connect>
       <Stream url="wss://api.pipecat.daily.co/ws/twilio">
         <Parameter name="_pipecatCloudServiceHost"
           value="AGENT_NAME.ORGANIZATION_NAME"/>
       </Stream>
     </Connect>
   </Response>

   Replace:

   • AGENT_NAME with the name of the agent you deployed to Pipecat Cloud
   • ORGANIZATION_NAME with the name of your Pipecat Cloud organization

   For the two benchmark agents, create two TwiML Bins — one with cekura-benchmark-appointments.ORGANIZATION_NAME, one with cekura-benchmark-insurance.ORGANIZATION_NAME.

5. Click Save

2. Assign TwiML Bin to Your Phone Number

1. Navigate to Phone Numbers → Manage → Active Numbers
2. Click on your Twilio phone number
3. In the "Voice Configuration" section:
   • Set "A call comes in" to TwiML Bin
   • Select the TwiML Bin you created
4. Click Save configuration

For the benchmark setup, repeat for both numbers: the appointments number gets the appointments TwiML Bin, the insurance number gets the insurance one. The dialed number fully determines which agent answers — there is no routing logic in the bot.

Server

1. Navigate to server directory:

   cd server

2. Install dependencies:

   uv sync

3. Configure environment variables:

   cp env.example .env
   # Edit .env and add your API keys

4. Run the bot (pick an agent definition via AGENT_DIR):

   AGENT_DIR=appointments uv run bot.py

   The runner serves every transport; the caller selects which one (a web/mobile client picks its transport when it connects; a telephony provider connects to /ws).

   For telephony, expose the bot with a public tunnel and point your provider's webhook at it:

   ngrok http 7860
   # then set the provider's webhook to wss://<your-ngrok-host>/ws

5. Test headless (no phone call) with the eval transport:

   AGENT_DIR=insurance uv run bot.py -t eval

   This boots the bot as a local eval WebSocket server — a clean boot confirms the agent dir loaded and all tools registered. Drive it with scripted scenarios via uv run pipecat eval run <scenario>.yaml (requires the pipecat-ai[evals] extra).

Project Structure

cekura-benchmark/
├── server/                          # Python bot server
│   ├── bot.py                       # Main bot implementation (agent-agnostic)
│   ├── appointments/                # Agent definition: clinic scheduling
│   ├── insurance/                   # Agent definition: Medicare insurance
│   ├── pyproject.toml               # Python dependencies
│   ├── env.example                  # Environment variables template
│   ├── .env                         # Your API keys (git-ignored)
│   ├── Dockerfile                   # Container image for Pipecat Cloud
│   ├── pcc-deploy-appointments.toml # Deploy config: appointments agent
│   ├── pcc-deploy-insurance.toml    # Deploy config: insurance agent
│   └── deploy.sh                    # Deploys both agents from one .env
├── .gitignore                       # Git ignore patterns
└── README.md                        # This file

Deploying to Pipecat Cloud

Both agents deploy from the same code with one command:

cd server
pipecat cloud auth login   # one-time
./deploy.sh

For each agent, the script generates that agent's secret set from the shared .env plus its single differing line (AGENT_DIR=appointments or AGENT_DIR=insurance), then deploys with the matching pcc-deploy-<agent>.toml. Pipecat Cloud's content-hash build cache reuses one image across both deploys, so the two agents run byte-identical code.

You can learn how to deploy to Pipecat Cloud in the Pipecat Quickstart Guide.

Refer to the Pipecat Cloud Documentation to learn more about configuring, deploying, and managing your agents in Pipecat Cloud.

Building with an AI coding agent

Extending this bot with Claude Code, Codex, or another AI coding assistant? Give it live, accurate Pipecat context instead of stale training data with the Pipecat Context Hub — a local index of Pipecat docs, examples, and API source your agent queries over MCP:

# Build the local index (first run takes a couple of minutes)
uvx pipecat-ai-context-hub@latest refresh

# Add it to your agent (use the line for the one you use)
claude mcp add pipecat-context-hub -- uvx pipecat-ai-context-hub serve   # Claude Code
codex mcp add pipecat-context-hub -- uvx pipecat-ai-context-hub serve    # Codex

MCP servers load at session start, so add it before opening your coding session. See the Pipecat Context Hub docs for the full setup.

Learn More

• Pipecat Documentation
• Pipecat GitHub
• Pipecat Examples
• Discord Community