Any way to add system messages mid-conversation?

AI SDK
1

Hey there,

I am building a conversational AI application where I need to provide the AI with some information during the conversation when something changes. For example if an event happens I want to let the AI know about it, it could be the user changes their name or something else. I have quite a few of these event. I tried to read throught the documentation but couldn’t find a solution. I tried to use the role: ‘system’, but that is only allowed at the beginning of the conversation for some reason.

Is there any solution to adding system messages during a conversation, that does not include just appending them to the users next request?
Thanks.

3

You can use the “developer” role which can go on any message throughout the conversation

However models tend to follow instructions best at either end of the prompt, so the beginning is still the best place for “system prompt” type instructions [2307.03172] Lost in the Middle: How Language Models Use Long Contexts

image
image1400×424 118 KB

So I’d recommend making your system prompt something variable that you can adjust as new information comes in. You could consider storing it in Redis and fetching on each message, then when an event happens you add the event info to the stored system prompt. Or you store a list of events and your system prompt is constructed by fetching them as bullet points

4

Thanks you for the response.

I did consider appending them to the original system prompt, just wasn’t how the model might react to updates being at the start of the conversation instead of at the end, when they cronologycally occured, but I will try it out.

I also didn’t know about the developer role. I don’t think conversation length will be an issue in this app, as conversations are quite small and simple, but I’ll try both approaches and see which one works best.

Thanks a lot!

5

@jacobparis Hey just checked and it doesn’t seem like the AI SDK supports the ‘developer’ role in UIMessage or in ModelMessage. Is there something I am missing? How do I set the role to developer without getting a type error?
Thanks