addToolResult and append causes weird flashes

Help
1

I’m working on a feature where my app will have multiple client-side tools. One of the client-side tools (tools without execute) is to allow user to choose a person from a list of presented users by AI with Person List as args (which my tool will render with)

Example of how everything would work:

  • User will say: Update the first name to user to “John”.
  • Bot: Sure, select a user from this list. (a client side tool which displays all people in UL/LI in HTML).
  • User: Types the “current old/name” (or clicks on that UL or LI).
  • Bot: Sure, it’s completed.

Current versus Expected behavior

Initially, I started using the addToolResult to complete the tool execution. But, that’s incomplete. I’d like to also post a message on “chat window” as soon as user interacts with any of these tools (in this example, selects a person).

Now, in order for me to also post on chat window, I started append() call in addition to addToResult(). And, that caused ChatWindow screen to show weird flash because “messages” array from useChat() has inconsistent responses.

Here’s my sample code:

const handleOnClick = async ({ person }) => {
  await chatContext?.addToolResult({
    toolCallId,
    result: { personId: person.id },
  });

  // Delay for 2 seconds
  await new Promise(resolve => setTimeout(resolve, 2000));

  await chatContext?.append({
    content: person.personDisplayName,
    role: 'user',
    createdAt: new Date(),
    id: Math.random().toString(),
  });
}

<ul>
  <li><Button onClick={handleOnClick} /></li>
  ...
</ul>

My questions:

  • What’s the right way to achieve this? I want to post a message on the chat window when user interacts with any client side tool, without causing any weird flash.
  • How do I complete tool execution by typing a message “without having to always call” addToolResult().?

@ai-sdk latest version.

2

You must call addToolResult to end the tool execution. If you want to be able to type a message, you should check if you’re in tool state and pass that to addToolResult INSTEAD of sending the message

You need to keep messages in sync between client/server so to avoid the flash you can either

  • add the message server-side when processing the tool result
  • or keep a separate array of tool results that you intersperse into the UI, separate from AI SDK

Here’s the official doc on adding Human in the Loop interactions to your chat

3

Yes, I’m aware about requirement to complete the tool execution BEFORE we can post a message. If you observe the following code, I’m already doing it where this code gets executed every time I post any message upon pressing Return key on text input.

let pendingToolInvocation: any;
messages.forEach(message => {
  // Find any pending tool call
  const toolPart: any = message.parts.find((part: any) => {
    return part.type === 'tool-invocation' && part.toolInvocation.state === 'call';
  });
  if (toolPart) {
    pendingToolInvocation = toolPart.toolInvocation;
  }
});

// Check any pending code
if (pendingToolInvocation) {
  addToolResult({
    toolCallId: pendingToolInvocation.toolCallId as string,
    result,
  });

  // Delay for 1 second
  await new Promise(resolve => setTimeout(resolve, 1000));
}

// Submits the message
onSubmit(e, currentMessage, currentFiles);

In fact, I have added a delay as well to let the call stabalize before I “send” message back to Server.

Any thoughts/advice on how to handle this better?

Are you saying, there’s absolutely no way to send “toolCall result” & “message” together (even with some delay)?