Structured outputs with AI Gateway

Help
1

Can’t see anything in the AI gateway docs on structured outputs, and when I try requesting structured outputs with the OpenAI compatible API it always fails.

Am I missing something, or is structured outputs not supported in the AI gateway? That would be a complete blocker for me using it if not..

3

Hi @palmwi2010, welcome to the Vercel Community!

I’m assuming you are using AI SDK with the AI Gateway. Have you given a try to the instructions in this documentation?

If so, can you share your code snippet and the errors you are getting?

4

Thanks for the quick response!

I’m actually not using the AI SDK - I’m just testing out the OpenAI compatible API for the AI gateway, to see if it could make sense for my Python application.

For structured outputs, I follow the same example as in the OpenAI API docs, but do not receive a JSON response, which then triggers a Pydantic vaidation error. The same code executes successfully using the OpenAI endpoint itself.

See code and error below.

async def test_structured_output():
    client = OpenAI(base_url='https://ai-gateway.vercel.sh/v1', api_key=os.getenv("VERCEL_API_KEY"))

    class CalendarEvent(BaseModel):
        name: str
        date: str
        participants: list[str]

    completion = client.chat.completions.parse(
        model="openai/gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        response_format=CalendarEvent,
    )

    event = completion.choices[0].message.parsed
    print(event)

Error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for CalendarEvent
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='Event: Science Fair  \nP...nd Bob  \nDay: Friday  ', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid
5

I see. I think your system prompt and class definition can use some more information. This will help the AI generate a good JSON response.

Maybe trying out the following system prompt will help:

You are an expert at structured data extraction. You will be given unstructured text and should convert it into the given structure with a valid JSON format.
6

Hi Anshuman,

This prompt example follows the exact example from the OpenAI docs, and executes fine when using the real OpenAI endpoint. But never successfully returns structured data when using the Vercel AI gateway endpoint. Therefore, I think there is a more fundamental issue with how structured output requests are passed on to the providers.

Example working with the OpenAI endpoint:

async def test_structured_output():
    client = OpenAI()

    class CalendarEvent(BaseModel):
        name: str
        date: str
        participants: list[str]

    completion = client.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        response_format=CalendarEvent,
    )

    event = completion.choices[0].message.parsed
    print(event)

Output:

name='Science Fair' date='Friday' participants=['Alice', 'Bob']
7

Thanks for providing an elaborate example. I’ll run this through our team. I’ll keep you posted here.

1 Like
9

Hi Anshuman,

Hope you’re well.

Just want to check if you are still looking into this?

Thanks!

10

Hi @palmwi2010, I was able to recreate the issue myself and our team is still investigating it. Sorry that it’s taking longer. I’ve just pinged them again. I’ll keep you posted here.

1 Like
11

Hi @palmwi2010, our team is working on improving the API for structured output params. In the meantime, you can use the following code when working with Pydantic for structured output:

from dotenv import load_dotenv
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.vercel import VercelProvider

load_dotenv()

class CityInfo(BaseModel):
    name: str
    country: str
    population: int
    famous_for: list[str]

agent = Agent(
    OpenAIModel('anthropic/claude-3.5-sonnet', provider=VercelProvider()),
    output_type=CityInfo,
    system_prompt='Provide accurate city information.'
)

if __name__ == '__main__':
    cities = ["Tokyo", "Paris", "New York"]
    
    print("=== PydanticAI x AI Gateway Structured Output ===\n")
    
    for city in cities:
        result = agent.run_sync(f'Tell me about {city}')
        info = result.output
        
        print(f"City: {info.name}")
        print(f"Country: {info.country}")
        print(f"Population: {info.population:,}")
        print(f"Famous for: {info.famous_for}")
        print("-" * 40)
1 Like
12

Great, thanks Anshuman for the help!

1 Like
13

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.