Is it possible to expose attachements to tools?

AI SDK
,
1

I’m trying to write a simple tool, that uses attachements uploaded by the user using the ai package.

But I can’t get it to work?


    const result = streamText({
        model: openai('gpt-4o'),
        messages,
        tools: {
            addUpdate: tool({
                description: 'A simple tool, that requires a user attachement',
                parameters: z.object({
                    name: z.string().min(1).max(200),
                    attachmentUrl: z.string().describe('Url of the attachment to add'),
                    // attachment: z
                    //     .object({
                    //         name: z.string().describe('The filename of the attachment'),
                    //         url: z.string().describe('The URL or data URL of the attachment'),
                    //         contentType: z.string().describe('The MIME type of the attachment'),
                    //     })
                }),

But it does not seem like the LLM can access user uploaded attachements? Is that true?

I noticed that if I access the messages

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": <some user message>
      },
      {
        "type": "image",
        "image": <image url>
      }
    ]
  },
1 Like
4

Looks like the current LLM setup doesn’t directly access uploaded attachments in messages—that’s a common limitation. You might need to handle the attachment URLs separately and pass them explicitly to your tools.
— For related coding tips, check classic code tool

1 Like
5

Did you figure out a solution? @alexbjorlig

6

Nope, did not find a solution.

7

I ran into a similar situation and noticed something in the docs that might help:

Context (experimental)
You can pass in arbitrary context from generateText or streamText via the experimental_context setting. This context is then available in the experimental_context tool execution option.

const result = await generateText({
  // ...
  tools: {
    someTool: tool({
      // ...
      execute: async (input, { experimental_context: context }) => {
        const typedContext = context as { example: string }; 
        // or use a type validation library
        // ...
      },
    }),
  },
  experimental_context: { example: '123' },
});

Have you tried using this approach?

8

Interesting - but no :thinking:

Will give it a try soon

1 Like