Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,8 +143,59 @@ GOOGLE_GEMINI_API_KEY=

# ELEVENLABS
ELEVENLABS_API_KEY=

# 60db (alternative voice provider) — see https://docs.60db.ai
# Set NEXT_PUBLIC_VOICE_PROVIDER=60db to route all voice calls through 60db
# instead of Vapi. Leave unset (or set to "vapi") to keep the original path.
NEXT_PUBLIC_VOICE_PROVIDER=
SIXTYDB_API_KEY=
SIXTYDB_API_BASE=https://api.60db.ai
NEXT_PUBLIC_SIXTYDB_LLM_MODEL=60db-tiny
NEXT_PUBLIC_SIXTYDB_DEFAULT_VOICE_ID=
```

## 60db Provider (alongside Vapi)

This repo ships a parallel **60db** voice path so the same UI can run on either
backend. Set `NEXT_PUBLIC_VOICE_PROVIDER=60db` and the app will route the call
through 60db's STT WebSocket + LLM chat completions + TTS WebSocket instead of
Vapi.

| File | Role |
|---|---|
| `hooks/useSixtyDb.ts` | Drop-in shape match for `useVapi` — same `{ status, messages, currentMessage, start, stop, ... }` surface |
| `lib/sixtydb/agent.ts` | Per-turn orchestrator: STT → LLM (with `searchBook` tool) → TTS, plus retrieval fallback |
| `lib/sixtydb/chat-client.ts` | `POST /v1/chat/completions` (model `60db-tiny`) with OpenAI-compatible `tool` pass-through |
| `lib/sixtydb/stt-socket.ts` | `wss://api.60db.ai/ws/stt` browser-mode (linear PCM 16k) |
| `lib/sixtydb/tts-socket.ts` | `wss://api.60db.ai/ws/tts` (LINEAR16 24k) |
| `lib/sixtydb/audio-pipeline.ts` | Mic capture (48k Float32 → 16k Int16) and Web Audio playback |
| `app/api/sixtydb/key/route.ts` | Clerk-gated server route returning the long-lived API key to the browser (60db has no token-mint endpoint yet) |
| `app/api/sixtydb/discovery/route.ts` | Server proxy for `/default-voices`, `/my-voices`, `/tts/models`, `/stt/models` |

**RAG flow** (matches Vapi's tool-call shape):
1. Every LLM turn defines a `searchBook` tool referencing the current book id.
2. If the model emits a tool call → run `searchBookSegments` (Mongo `$text`), feed
the result back, re-call the model.
3. If the model didn't call the tool but the user clearly asked a content
question and the reply looks evasive → inject the top-3 segments and retry.

**Security:** `/api/sixtydb/key` ships the long-lived `SIXTYDB_API_KEY` to the
browser (gated by Clerk auth). When 60db exposes a token-mint endpoint, swap
this for a short-lived token.

**Activation diff:**

```env
- NEXT_PUBLIC_VOICE_PROVIDER=
+ NEXT_PUBLIC_VOICE_PROVIDER=60db
+ SIXTYDB_API_KEY=sk_live_...
```

No code changes — `components/VapiControls.tsx` picks the hook from
`VOICE_PROVIDER` at module init, and `components/VoiceSelector.tsx` fetches the
60db voice catalog at runtime via the discovery proxy.


Replace the placeholder values with your real credentials. You can get these by signing up at: [**Clerk**](https://clerk.com), [**Vercel**](https://vercel.com), [**MongoDB**](https://www.mongodb.com), [**Vapi**](https://vapi.ai), [**Google AI Studio**](https://aistudio.google.com), [**ElevenLabs**](https://elevenlabs.io).

**Running the Project**
Expand Down
63 changes: 63 additions & 0 deletions app/api/sixtydb/discovery/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
import { NextResponse } from 'next/server';
import { auth } from '@clerk/nextjs/server';

/**
* Server-side proxy for 60db discovery endpoints. Keeps the API key off
* the wire so the browser can populate the voice / model pickers safely.
*
* GET /api/sixtydb/discovery?kind=default-voices -> /default-voices
* GET /api/sixtydb/discovery?kind=my-voices -> /my-voices
* GET /api/sixtydb/discovery?kind=tts-models -> /tts/models
* GET /api/sixtydb/discovery?kind=stt-models -> /stt/models
*/
const PATH_MAP: Record<string, string> = {
'default-voices': '/default-voices',
'my-voices': '/my-voices',
'tts-models': '/tts/models',
'stt-models': '/stt/models',
};

export async function GET(request: Request) {
const { userId } = await auth();
if (!userId) {
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}

const apiKey = process.env.SIXTYDB_API_KEY;
if (!apiKey) {
return NextResponse.json(
{ error: 'SIXTYDB_API_KEY not set on the server.' },
{ status: 500 },
);
}

const kind = new URL(request.url).searchParams.get('kind') ?? '';
const path = PATH_MAP[kind];
if (!path) {
return NextResponse.json(
{
error: `Unknown kind '${kind}'. Use one of: ${Object.keys(PATH_MAP).join(', ')}.`,
},
{ status: 400 },
);
}

const apiBase = (process.env.SIXTYDB_API_BASE || 'https://api.60db.ai').replace(/\/$/, '');

try {
const res = await fetch(`${apiBase}${path}`, {
headers: { Authorization: `Bearer ${apiKey}` },
cache: 'no-store',
});
const body = await res.text();
return new NextResponse(body, {
status: res.status,
headers: { 'Content-Type': res.headers.get('Content-Type') ?? 'application/json' },
});
} catch (err) {
return NextResponse.json(
{ error: `60db ${path} request failed: ${(err as Error).message}` },
{ status: 502 },
);
Comment on lines +47 to +61

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | ⚡ Quick win

Add an explicit timeout to the upstream discovery fetch.

At Line 48, fetch has no timeout. If 60db stalls, this route can hang until platform-level timeout, tying up request capacity.

Suggested patch
 export async function GET(request: Request) {
@@
-    try {
-        const res = await fetch(`${apiBase}${path}`, {
-            headers: { Authorization: `Bearer ${apiKey}` },
-            cache: 'no-store',
-        });
-        const body = await res.text();
-        return new NextResponse(body, {
-            status: res.status,
-            headers: { 'Content-Type': res.headers.get('Content-Type') ?? 'application/json' },
-        });
-    } catch (err) {
+    try {
+        const controller = new AbortController();
+        const timeout = setTimeout(() => controller.abort(), 8000);
+        try {
+            const res = await fetch(`${apiBase}${path}`, {
+                headers: { Authorization: `Bearer ${apiKey}` },
+                cache: 'no-store',
+                signal: controller.signal,
+            });
+            const body = await res.text();
+            return new NextResponse(body, {
+                status: res.status,
+                headers: { 'Content-Type': res.headers.get('Content-Type') ?? 'application/json' },
+            });
+        } finally {
+            clearTimeout(timeout);
+        }
+    } catch (err) {
+        if ((err as Error).name === 'AbortError') {
+            return NextResponse.json(
+                { error: `60db ${path} request timed out.` },
+                { status: 504 },
+            );
+        }
         return NextResponse.json(
             { error: `60db ${path} request failed: ${(err as Error).message}` },
             { status: 502 },
         );
     }
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/api/sixtydb/discovery/route.ts` around lines 47 - 61, The fetch to
`${apiBase}${path}` has no timeout; wrap the request in an AbortController
(create controller, pass signal to fetch) and set a timer (e.g. from a new
DISCOVERY_FETCH_TIMEOUT_MS env var or a constant) that calls controller.abort()
after the timeout, then clear the timer on success; update the catch to detect
abort errors and return an appropriate 504/timeout message via NextResponse.json
(keep existing error format), referencing the fetch call, apiBase, path,
AbortController signal, and NextResponse.json so you can locate and modify the
code.

}
}
27 changes: 27 additions & 0 deletions app/api/sixtydb/key/route.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
import { NextResponse } from 'next/server';
import { auth } from '@clerk/nextjs/server';

/**
* Returns the 60db API key to the browser so it can open the STT/TTS
* WebSockets. 60db has no token-mint endpoint yet, so this exposes the
* long-lived key. Gated behind Clerk auth so only signed-in users hit it.
*/
export async function POST() {
const { userId } = await auth();
if (!userId) {
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
}

const apiKey = process.env.SIXTYDB_API_KEY;
if (!apiKey) {
return NextResponse.json(
{ error: 'SIXTYDB_API_KEY not set on the server.' },
{ status: 500 },
);
}

return NextResponse.json({
api_key: apiKey,
api_base: process.env.SIXTYDB_API_BASE || 'https://api.60db.ai',
});
Comment on lines +23 to +26

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical | 🏗️ Heavy lift

Do not return the long-lived SIXTYDB_API_KEY to the browser.

At Line 24, this endpoint returns a reusable server secret to clients. Any signed-in user can exfiltrate and use it off-platform, bypassing your app’s controls and potentially incurring uncontrolled spend.

Use a server-side WebSocket proxy (or provider-issued short-lived token when available) so the long-lived key never leaves server trust boundaries.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@app/api/sixtydb/key/route.ts` around lines 23 - 26, The route is returning
the long-lived SIXTYDB_API_KEY to clients via NextResponse.json (the api_key
variable) — remove the api_key field from the response and never expose
process.env.SIXTYDB_API_KEY in route.ts; instead keep only non-secret metadata
(e.g., api_base) or nothing. Implement a server-side proxy endpoint (or use
provider-issued short-lived tokens when available) that forwards client requests
to 60db using the server-side secret, or add a server-only route that mints
short-lived tokens and returns those, and update any client code to call that
proxy/mint endpoint instead of reading api_key from this route.

}
9 changes: 8 additions & 1 deletion components/VapiControls.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import {Mic, MicOff} from "lucide-react";
import useVapi from "@/hooks/useVapi";
import useSixtyDb from "@/hooks/useSixtyDb";
import { VOICE_PROVIDER } from "@/lib/constants";
import {IBook} from "@/types";
import Image from "next/image";
import Transcript from "@/components/Transcript";
Expand All @@ -10,8 +12,13 @@ import {toast} from "sonner";
import {useRouter} from "next/navigation";
import {useEffect} from "react";

// Pick the agent transport at module init — env-level decision, never
// flips at runtime. The two hooks expose the same shape so the rest of
// this component is provider-agnostic.
const useVoiceAgent = VOICE_PROVIDER === '60db' ? useSixtyDb : useVapi;

const VapiControls = ({ book }: { book: IBook }) => {
const { status, isActive, messages, currentMessage, currentUserMessage, duration, start, stop, clearError, limitError, isBillingError, maxDurationSeconds } = useVapi(book)
const { status, isActive, messages, currentMessage, currentUserMessage, duration, start, stop, clearError, limitError, isBillingError, maxDurationSeconds } = useVoiceAgent(book)
const router = useRouter();

useEffect(() => {
Expand Down
Loading