Is it possible to expose attachements to tools?

(edited)

I'm trying to write a simple tool, that uses attachements uploaded by the user using the ai package.

But I can't get it to work?


    const result = streamText({
        model: openai('gpt-4o'),
        messages,
        tools: {
            addUpdate: tool({
                description: 'A simple tool, that requires a user attachement',
                parameters: z.object({
                    name: z.string().min(1).max(200),
                    attachmentUrl: z.string().describe('Url of the attachment to add'),
                    // attachment: z
                    //     .object({
                    //         name: z.string().describe('The filename of the attachment'),
                    //         url: z.string().describe('The URL or data URL of the attachment'),
                    //         contentType: z.string().describe('The MIME type of the attachment'),
                    //     })
                }),

But it does not seem like the LLM can access user uploaded attachements? Is that true?

I noticed that if I access the messages

[
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": <some user message>
      },
      {
        "type": "image",
        "image": <image url>
      }
    ]
  },

Looks like the current LLM setup doesn’t directly access uploaded attachments in messages—that’s a common limitation. You might need to handle the attachment URLs separately and pass them explicitly to your tools. — For related coding tips, check classic code tool

Did you figure out a solution?

Nope, did not find a solution.

I ran into a similar situation and noticed something in the docs that might help:

Context (experimental) You can pass in arbitrary context from generateText or streamText via the experimental_context setting. This context is then available in the experimental_context tool execution option.

const result = await generateText({
  // ...
  tools: {
    someTool: tool({
      // ...
      execute: async (input, { experimental_context: context }) => {
        const typedContext = context as { example: string }; 
        // or use a type validation library
        // ...
      },
    }),
  },
  experimental_context: { example: '123' },
});

Have you tried using this approach?

Interesting - but no 🤔

Will give it a try soon

This topic is closed. It no longer accepts new replies.

const result = streamText({ model: openai('gpt-4o'), messages, tools: { addUpdate: tool({ description: 'A simple tool, that requires a user attachement', parameters: z.object({ name: z.string().min(1).max(200), attachmentUrl: z.string().describe('Url of the attachment to add'), // attachment: z // .object({ // name: z.string().describe('The filename of the attachment'), // url: z.string().describe('The URL or data URL of the attachment'), // contentType: z.string().describe('The MIME type of the attachment'), // }) }),

[ { "role": "user", "content": [ { "type": "text", "text": <some user message> }, { "type": "image", "image": <image url> } ] },

const result = await generateText({ // ... tools: { someTool: tool({ // ... execute: async (input, { experimental_context: context }) => { const typedContext = context as { example: string }; // or use a type validation library // ... }, }), }, experimental_context: { example: '123' }, });