Skip to content

Unable to pass PDF files from the tool calls #8014

@janostik

Description

@janostik

Description

When creating a toolcall which should fetch a specific PDF file and make it available to a model I'm facing several issues. Sample implementation:

import { Storage } from '@google-cloud/storage';
import { tool } from 'ai';
import { z } from 'zod';
import { generateUUID } from '../utils';

const storage = new Storage();
const bucketName = process.env.GOOGLE_STORAGE_BUCKET || 'prodeen-tmp';

export const fetchFile = () => {
  return tool({
    description: 'Fetch a file based on it's name',
    inputSchema: z.object({
      name: z.string().describe('Name o the remote file')
    }),
    execute: async ({ name }) => {
      // Logic of fetching the file
      // ...
      // Download saved file from remote storage and convert to base64
      const [downloadedBuffer] = await file.download();
      const base64 = downloadedBuffer.toString('base64');

      return {
        name: displayName,
        type: 'pdf',
        data: base64,
        text: '',
      };
    },
    toModelOutput: (result) => {
      const { type, data, text } = result as {
        type: string;
        data: string;
        text: string;
      };
      switch (type) {
        case 'text':
          return { type: 'text', value: text };
        case 'pdf':
          return {
            type: 'content',
            value: [
              {
                type: 'media',
                data,
                mediaType: 'application/pdf',
              },
            ],
          };
        default:
          return { type: 'error-text', value: 'Unsupported file type' };
      }
    },
  });
};

When using Gemini models, I do not end up with error, but the model is not able to read the file. If I use anthropic models I end up with following error:

'media type: application/pdf' functionality not supported.

even though I'm using model which supports reading PDF files.

Documentation on this is not clear and only provides use cases with Images. Is it possible to pass the PDF files from tools as well? Also is it needed to use the base64 string? In the types I could not pass the url without having typescript issues.

I'm running on bun.

AI SDK Version

  • "ai": "5.0.0",
  • "@ai-sdk/anthropic": "^2.0.0",
  • "@ai-sdk/google": "2.0.0",

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions