-
Notifications
You must be signed in to change notification settings - Fork 3.1k
Open
Description
Description
When creating a toolcall which should fetch a specific PDF file and make it available to a model I'm facing several issues. Sample implementation:
import { Storage } from '@google-cloud/storage';
import { tool } from 'ai';
import { z } from 'zod';
import { generateUUID } from '../utils';
const storage = new Storage();
const bucketName = process.env.GOOGLE_STORAGE_BUCKET || 'prodeen-tmp';
export const fetchFile = () => {
return tool({
description: 'Fetch a file based on it's name',
inputSchema: z.object({
name: z.string().describe('Name o the remote file')
}),
execute: async ({ name }) => {
// Logic of fetching the file
// ...
// Download saved file from remote storage and convert to base64
const [downloadedBuffer] = await file.download();
const base64 = downloadedBuffer.toString('base64');
return {
name: displayName,
type: 'pdf',
data: base64,
text: '',
};
},
toModelOutput: (result) => {
const { type, data, text } = result as {
type: string;
data: string;
text: string;
};
switch (type) {
case 'text':
return { type: 'text', value: text };
case 'pdf':
return {
type: 'content',
value: [
{
type: 'media',
data,
mediaType: 'application/pdf',
},
],
};
default:
return { type: 'error-text', value: 'Unsupported file type' };
}
},
});
};
When using Gemini models, I do not end up with error, but the model is not able to read the file. If I use anthropic models I end up with following error:
'media type: application/pdf' functionality not supported.
even though I'm using model which supports reading PDF files.
Documentation on this is not clear and only provides use cases with Images. Is it possible to pass the PDF files from tools as well? Also is it needed to use the base64 string? In the types I could not pass the url without having typescript issues.
I'm running on bun.
AI SDK Version
- "ai": "5.0.0",
- "@ai-sdk/anthropic": "^2.0.0",
- "@ai-sdk/google": "2.0.0",