Structured outputs with AI Gateway

Can't see anything in the AI gateway docs on structured outputs, and when I try requesting structured outputs with the OpenAI compatible API it always fails.

Am I missing something, or is structured outputs not supported in the AI gateway? That would be a complete blocker for me using it if not..

Vercel Staff

Hi , welcome to the Vercel Community!

I'm assuming you are using AI SDK with the AI Gateway. Have you given a try to the instructions in this documentation?

If so, can you share your code snippet and the errors you are getting?

Thanks for the quick response!

I’m actually not using the AI SDK - I’m just testing out the OpenAI compatible API for the AI gateway, to see if it could make sense for my Python application.

For structured outputs, I follow the same example as in the OpenAI API docs, but do not receive a JSON response, which then triggers a Pydantic vaidation error. The same code executes successfully using the OpenAI endpoint itself.

See code and error below.

async def test_structured_output():
    client = OpenAI(base_url='https://ai-gateway.vercel.sh/v1', api_key=os.getenv("VERCEL_API_KEY"))

    class CalendarEvent(BaseModel):
        name: str
        date: str
        participants: list[str]

    completion = client.chat.completions.parse(
        model="openai/gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        response_format=CalendarEvent,
    )

    event = completion.choices[0].message.parsed
    print(event)

Error:

pydantic_core._pydantic_core.ValidationError: 1 validation error for CalendarEvent
  Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='Event: Science Fair  
P...nd Bob  
Day: Friday  ', input_type=str]
    For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

Vercel Staff

I see. I think your system prompt and class definition can use some more information. This will help the AI generate a good JSON response.

Maybe trying out the following system prompt will help:

You are an expert at structured data extraction. You will be given unstructured text and should convert it into the given structure with a valid JSON format.

Hi Anshuman,

This prompt example follows the exact example from the OpenAI docs, and executes fine when using the real OpenAI endpoint. But never successfully returns structured data when using the Vercel AI gateway endpoint. Therefore, I think there is a more fundamental issue with how structured output requests are passed on to the providers.

Example working with the OpenAI endpoint:

async def test_structured_output():
    client = OpenAI()

    class CalendarEvent(BaseModel):
        name: str
        date: str
        participants: list[str]

    completion = client.chat.completions.parse(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Extract the event information."},
            {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."},
        ],
        response_format=CalendarEvent,
    )

    event = completion.choices[0].message.parsed
    print(event)

Output:

name='Science Fair' date='Friday' participants=['Alice', 'Bob']

Vercel Staff

Thanks for providing an elaborate example. I'll run this through our team. I'll keep you posted here.

reply to #7

Hi Anshuman,

Hope you’re well.

Just want to check if you are still looking into this?

Thanks!

Vercel Staff

Hi , I was able to recreate the issue myself and our team is still investigating it. Sorry that it's taking longer. I've just pinged them again. I'll keep you posted here.

Vercel Staff

Hi , our team is working on improving the API for structured output params. In the meantime, you can use the following code when working with Pydantic for structured output:

from dotenv import load_dotenv
from pydantic import BaseModel
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIModel
from pydantic_ai.providers.vercel import VercelProvider

load_dotenv()

class CityInfo(BaseModel):
    name: str
    country: str
    population: int
    famous_for: list[str]

agent = Agent(
    OpenAIModel('anthropic/claude-3.5-sonnet', provider=VercelProvider()),
    output_type=CityInfo,
    system_prompt='Provide accurate city information.'
)

if __name__ == '__main__':
    cities = ["Tokyo", "Paris", "New York"]
    
    print("=== PydanticAI x AI Gateway Structured Output ===
")
    
    for city in cities:
        result = agent.run_sync(f'Tell me about {city}')
        info = result.output
        
        print(f"City: {info.name}")
        print(f"Country: {info.country}")
        print(f"Population: {info.population:,}")
        print(f"Famous for: {info.famous_for}")
        print("-" * 40)

Great, thanks Anshuman for the help!

This topic is closed. It no longer accepts new replies.

async def test_structured_output(): client = OpenAI(base_url='https://ai-gateway.vercel.sh/v1', api_key=os.getenv("VERCEL_API_KEY")) class CalendarEvent(BaseModel): name: str date: str participants: list[str] completion = client.chat.completions.parse( model="openai/gpt-4o-mini", messages=[ {"role": "system", "content": "Extract the event information."}, {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, ], response_format=CalendarEvent, ) event = completion.choices[0].message.parsed print(event)

pydantic_core._pydantic_core.ValidationError: 1 validation error for CalendarEvent Invalid JSON: expected value at line 1 column 1 [type=json_invalid, input_value='Event: Science Fair P...nd Bob Day: Friday ', input_type=str] For further information visit https://errors.pydantic.dev/2.11/v/json_invalid

async def test_structured_output(): client = OpenAI() class CalendarEvent(BaseModel): name: str date: str participants: list[str] completion = client.chat.completions.parse( model="gpt-4o-mini", messages=[ {"role": "system", "content": "Extract the event information."}, {"role": "user", "content": "Alice and Bob are going to a science fair on Friday."}, ], response_format=CalendarEvent, ) event = completion.choices[0].message.parsed print(event)

from dotenv import load_dotenv from pydantic import BaseModel from pydantic_ai import Agent from pydantic_ai.models.openai import OpenAIModel from pydantic_ai.providers.vercel import VercelProvider load_dotenv() class CityInfo(BaseModel): name: str country: str population: int famous_for: list[str] agent = Agent( OpenAIModel('anthropic/claude-3.5-sonnet', provider=VercelProvider()), output_type=CityInfo, system_prompt='Provide accurate city information.' ) if __name__ == '__main__': cities = ["Tokyo", "Paris", "New York"] print("=== PydanticAI x AI Gateway Structured Output === ") for city in cities: result = agent.run_sync(f'Tell me about {city}') info = result.output print(f"City: {info.name}") print(f"Country: {info.country}") print(f"Population: {info.population:,}") print(f"Famous for: {info.famous_for}") print("-" * 40)