Your approach is solid! You’re on the right track with using prepareStep and tracking compaction state on the frontend. Let me address your questions:
1. Is this the right approach?
Yes, this is a good approach. There’s no built-in conversation compaction in AI SDK v5, so managing it yourself is the way to go. Your solution of tracking compaction state on the frontend is smart.
2. Does prepareStep work with streamText?
Yes, prepareStep works with both generateText and streamText. Here’s how it looks with streamText:
// app/api/chat/route.ts import { streamText } from 'ai' import { openai } from '@ai-sdk/openai' export async function POST(req: Request) { const { messages, conversationSummary, compactedUpToIndex } = await req.json() const result = streamText({ model: openai('gpt-4'), messages, prepareStep: async ({ messages }) => { // Your compaction logic here if (conversationSummary) { return { system: `${systemPrompt}\n\n[SUMMARY]\n${conversationSummary}`, messages: messages.slice(compactedUpToIndex), } } if (messages.length >= 150) { // Generate summary and handle compaction const summary = await generateSummary(messages.slice(0, -20)) return { system: `${systemPrompt}\n\n[SUMMARY]\n${summary}`, messages: messages.slice(-20), } } return {} }, }) return result.toDataStreamResponse() }
3. Alternative approach - Frontend slicing
Actually, I’d recommend a simpler approach: let the frontend handle the slicing. This avoids the complexity of streaming compaction data back:
// Frontend component 'use client' const [compactionState, setCompactionState] = useState({ summary: null, compactedUpToIndex: 0, }) const { messages, input, handleInputChange, handleSubmit } = useChat({ api: '/api/chat', body: { conversationSummary: compactionState.summary, compactedUpToIndex: compactionState.compactedUpToIndex, }, onFinish: async (message) => { // Check if we need to compact after this response if (messages.length >= 150 && !compactionState.summary) { const summaryResponse = await fetch('/api/summarize', { method: 'POST', body: JSON.stringify({ messages: messages.slice(0, -20) }), }) const { summary } = await summaryResponse.json() setCompactionState({ summary, compactedUpToIndex: messages.length - 20, }) } }, }) // Send only relevant messages to the API const messagesToSend = compactionState.summary ? messages.slice(compactionState.compactedUpToIndex) : messages
Then your API route becomes much simpler:
// app/api/chat/route.ts export async function POST(req: Request) { const { messages, conversationSummary } = await req.json() const systemMessage = conversationSummary ? `${systemPrompt}\n\n[SUMMARY]\n${conversationSummary}` : systemPrompt const result = streamText({ model: openai('gpt-4'), system: systemMessage, messages, }) return result.toDataStreamResponse() }
4. Common patterns
The pattern you’re using is solid. Many developers handle this by:
- Keeping full conversation history in UI state
- Tracking compaction metadata separately
- Only sending relevant context to the LLM
- Periodically summarizing older messages
Your approach avoids re-compaction and maintains UI consistency, which is exactly what you want.