Vercel AI Gateway latency and provider routing issues with generateObject

Hi there,

For a few months now, I was using the Vercel AI Gateway to generate JSON objects with a zod schema. I am currently using AI SDK@5.0.27 with the gpt-oss-20, gpt-oss-120, and haiku-4.5 models.

Current Behavior

  • Latency: Response times usually were fast (3-4s), but since yesterday, providers started to peak at 45-60s for a single call.
  • Failures: Calls are failing to generate the object after the long wait.
  • Provider Mismatch: The AI Gateway dashboard reports that the used providers are fireworks, but the logs of the SDK indicate it’s using bedrock.
  • If I restrict the providers to ones without latency issues, like groq:
    const res = await generateObject({
      // ...
      providerOptions: {
        gateway: {
          only: ['groq', 'anthropic'],
        },
      },
    });
    
    I still get errors about JSON that could not get generated: Failed to validate JSON. Please adjust your prompt. See 'failed_generation' for more details.. It does not give me a failed_generation object.
  • I tried to switch to another model openai/gpt-5-nano, but the results were similar for the providers openai and azure: Invalid schema for response_format 'response': schema must be a JSON Schema of 'type: \"object\"', got 'type: \"None\"'.

Questions

Before I go into detail about what exactly is failing:

  1. Are there any known issues because of the current dxb1 downtime?
  2. Is anyone else experiencing this?
  3. I tried to upgrade to v6, but it would require rebuilding my entire integration due to generateObject deprecation, so I’m not sure if that’s the reason.
  4. Is there anything else that could cause this? Given the fact I did change nothing about my implementation

Thanks!

Hi @madebyfabian, welcome to the Vercel Community!

I’m sorry that you’re facing degraded performance. Could you confirm when you first noticed it?

Also, could you share a minimal code that you’re able to recreate the JSON failure or 45-60s response times for? I will try to use the same code to recreate it with my API key.

While, I don’t recommend it, the migration won’t be difficult because I did it for one of our projects and the generateText with schema works identically to generateObject.

Thanks!
It seems that it first appeared yesterday, March 2nd, 17:00 UTC. Before that, everything was working.

I tried to create a minimal reproduction. Original prompt is longer and the input is just demo. The task is to translate and keep the JSON structure.
While this is not taking as long (only about 15 seconds, but that’s still too much), it’s also failing with the same error:

Reproduction Code
export async function GET() {
	const systemPrompt = `
		<context_gathering>
			Du bist ein professioneller Ăśbersetzer fĂĽr Krypto-News-Zusammenfassungen. 
			Du bekommst eine Zusammenfassung der wichtigsten News und eine Analyse und musst diese ĂĽbersetzen.
			Beachte dabei alle Regeln, welche definiert sind.
			Die Zielsprache ist Deutsch.
		</context_gathering>
	`

	const input = {
		'news': [
			{
				'ai_wrapup_text':
					'HYPE rose about 6% as Iran‑related tensions rattled markets, even as Bitcoin and other risk assets fell and most traditional venues were closed.  Hyperliquid, an on‑chain decentralized perpetuals exchange, absorbed early weekend volume and became a key venue for traders looking to express risk during the shutdown.\n\nTrading volume on the platform reached a near one‑month high of roughly $200 million, and HYPE’s price surged to about $32 as fee‑generated revenue and token buybacks tied directly to that activity boosted demand.  The token is up around 25 % year‑to‑date, though it remains below its September peak near $58.\n\nAnalysts say that always‑on, non‑custodial platforms like Hyperliquid may become first‑response venues for geopolitical risk, turning weekend crises into a structural tailwind for trading volume, fee revenue, and HYPE token demand.',
				'description':
					"As tensions escalated over Iran-related headlines this weekend, Hyperliquid's HYPE token rose about 6% as traders turned to the always-on decentralized perpetuals platform to express risk while many traditional markets were closed. Bitcoin and other risk assets fell as Iran-related tensions escalated, while oil and gold moved higher amid a broader risk-off shift.",
				'id': '903b420b-713c-490a-b9fe-a1cdb2be5f22',
				'title': "Hyperliquid's Token Rises as Weekend Iran Shock Finds Few Open Markets",
			},
			{
				'ai_wrapup_text':
					'Bitcoin fell to $66,702 in early Monday trading, down 1.1% after a brief Sunday rally, as markets priced in the U.S.–Iran conflict and a 6% surge in oil prices. Brent crude rose as much as 13% at the open, settling around $77.50 amid concerns that the Strait of Hormuz is effectively closed, tightening risk‑asset sentiment. \n\nEthereum, Solana and XRP also slipped, with Solana dropping 8.1% over the week and leading losses among majors. Some traders argue downside risk is limited because the world has largely weaned itself from Iranian oil; however, price moves will hinge on whether the Strait reopens and how long Trump’s objectives take to materialize.',
				'description':
					"Bitcoin fell back to $66,700 as traditional markets opened to their first chance to price the weekend's military escalation, with oil surging to $77 and Asian equities dropping 1.4%.",
				'id': 'c341fa06-a8eb-4ed3-877d-25358151d98d',
				'title': 'Bitcoin, cryptos under pressure as oil spikes 6% and global markets price in U.S.-Iran conflict',
			},
			{
				'ai_wrapup_text':
					'U.S.-listed spot bitcoin and ether ETFs have seen record outflows in the last four months, with investors withdrawing $6.39bn from bitcoin and $2.76bn from ether, the longest losing streak since the funds launched in January 2024.\n\nThe massive exits highlight a collapse in institutional appetite for digital assets, coinciding with the sharp price drop of Bitcoin and Ether—Bitcoin fell from >$126,000 to ~$67,000, while Ether slid over 60% from August highs. 2024 saw initial inflows linked to pro‑crypto politics, but the October crash, partly driven by Binance pricing, drained momentum.',
				'description': 'Record outflows indicate that institutional appetite for digital assets has collapsed.',
				'id': '5e601588-d7d6-4cf3-a45b-35e8d632ca95',
				'title': 'Over $9 billion flees bitcoin and ether ETFs in four months',
			},
			{
				'ai_wrapup_text':
					'MicroStrategy, the largest corporate Bitcoin holder, raised the dividend on its STRC preferred stock to 11.50 % for March 2026, a 25‑basis‑point increase and the seventh hike since July 2025. The change, announced by Chairman Michael Saylor on X, keeps the shares near their $100 par value and is aimed at limiting volatility as Bitcoin’s drawdown continues to weigh on the company’s common shares.\n\nThe move comes as MSTR’s common shares have fallen 14.8 % year‑to‑date amid a 24 % slide in Bitcoin, while STRC shares remain relatively stable. CEO Phong Le said the firm plans to rely more on preferred‑share issuance for Bitcoin purchases, offering a lower‑volatility capital‑raising vehicle than diluting common equity. The strategy continues to acquire Bitcoin despite $6.6 billion in paper losses.',
				'description':
					'Strategy, formerly MicroStrategy, raised its STRC preferred stock dividend by 25 basis points for March 2026, as Bitcoin (BTC) drawdown continues to push MSTR shares down.',
				'id': '7cad5674-bbe2-48c3-924b-766357526bdd',
				'title': 'MicroStrategy Boosts STRC Dividend to 11.50% as Bitcoin Drawdown Pressures MSTR',
			},
			{
				'ai_wrapup_text':
					'XRP holders have moved more than $650 million worth of XRP onto Binance in the past week, according to on‑chain data from Darkfost and CryptoQuant. The shift follows a joint US‑Israel strike on Iran and subsequent reports of Iran’s leadership being targeted, which have spurred a broader sell‑off in risk assets.\n\nWhile large inflows to exchanges can signal potential selling pressure, analysts say the move could also reflect liquidity repositioning or precautionary hedges. XRP has fallen over 4 % in 24 hours, trading near $1.37, but it remains unclear whether the inflows presage a sustained distribution or a temporary reaction.',
				'description':
					'XRP (XRP) holders appear to be adopting a defensive stance amid intensifying geopolitical tensions between the United States, Israel, and Iran.',
				'id': '95dcf731-2167-4d72-8aa3-ee7d9b0a9708',
				'title': 'XRP Faces $650 Million Selling Pressure as Geopolitical Tensions Escalate',
			},
		],
	}

	const res = await generateObject({
		model: 'openai/gpt-oss-20b',
		messages: [
			{
				role: 'user',
				content: systemPrompt,
			},
			{
				role: 'user',
				content: JSON.stringify(input),
			},
		],
		schema: z.object({
			news: z.array(
				z.object({
					ai_wrapup_text_de: z.string(),
					description_de: z.string(),
					id: z.string(),
					title_de: z.string(),
				})
			),
		}),
	})

	return Response.json({ res })
}

Error:

Error
Error [AI_NoObjectGeneratedError]: No object generated: response did not match schema.
    at async GET (app/api/test/route.ts:61:14)
  59 |  }
  60 |
> 61 |  const res = await generateObject({
     |              ^
  62 |          model: 'openai/gpt-oss-20b',
  63 |          messages: [
  64 |                  { {
  text: '{"news":[{"title_de":"Hyperliquids Token steigt, während der Iran-Schock am Wochenende kaum offene Märkte findet","ai_wrapup_text_de":"HYPE stieg um etwa 6 % als iranspezifische Spannungen die Märkte erschütterten, während Bitcoin und andere Risikoinvestments fielen und die meisten traditionellen Märkte geschlossen waren. Hyperliquid, eine on‑Chain dezentrale Perpetuals-Börse, erfasste das frühe Wochenendvolumen und wurde zu einer wichtigen Plattform für Händler, die ihr Risiko während der Schließung ausdrücken wollten. Das Handelsvolumen auf der Plattform erreichte fast ein einmonatiges Hoch von rund 200 Mrd. USD, und der Preis von HYPE sprang auf etwa 32 USD, da gebührenbasierte Einnahmen und Token‑Rückkäufe, die direkt mit dieser Aktivität gekoppelt sind, die Nachfrage ankurbelt. Der Token ist Jahr‑zu‑Datum um rund 25 % gestiegen, bleibt jedoch unter seinem Septemberspike von ca. 58 USD. Analysten zufolge könnten immer online, nicht‑Kustodial-Plattformen wie Hyperliquid zu ersten Reaktionsplattformen für geopolitische Risiken werden und Wochenend‑Krisen in einen strukturellen Rückenwind für Handelsvolumen, Gebühren‑Einnahmen und HYPE‑Token‑Nachfrage verwandeln.","id":"903b420b-713c-490a-b9fe-a1cdb2be5f22","description_de":"Als sich die Spannungen über iranspezifische Schlagzeilen in diesem Wochenende zuspitzten, stieg Hyperliquids HYPE‑Token um etwa 6 %, da Händler zur immer‑Online dezentralen Perpetuals-Plattform wechselten, um ihr Risiko auszudrücken, während viele traditionelle Märkte geschlossen waren. Bitcoin und andere Risikoinvestments fielen mit steigenden Iran‑Spannungen, während Öl und Gold im Rahmen eines breiteren Risiko‑Abstiegs anstiegen."},{"title_de":"Bitcoin und Kryptowährungen unter Druck, während Öl um 6 % steigt und globale Märkte das US‑Iran-Konfliktpreiszenario einbeziehen","ai_wrapup_text_de":"Bitcoin fiel früh am Montag auf 66.702 USD, ein Rückgang von 1,1 %, nach einer kurzen Sonntags‑Rallye, da die Märkte das US‑Iran-Konflikt und einen Ölpreisanstieg von 6 % preisgaben. Brent‑Crude stieg zu Beginn um bis zu 13 % und schloss bei rund 77,50 USD, während die Befürchtung, dass der Hohrizum‑Kanal effektiv verschlossen sei, die Stimmung gegenüber Risikoinvestments dämpfte. Ethereum, Solana und XRP schüttelten sich ebenfalls ab; Solana hatte in der Woche 8,1 % verloren und war mit der größten Verlustquote unter den Hauptakteuren. Einige Händler argumentieren, dass das Abwärtsrisiko begrenzt sei, weil die Welt weitgehend von iranischem Öl abgeknöpft sei; die Preisbewegungen hängen jedoch davon ab, ob der Hohrizum‑Kanal wieder öffnet und wie lange Trumps Ziele dauern werden.","id":"c341fa06-a8eb-4ed3-877d-25358151d98d","description_de":"Bitcoin sank auf 66.700 USD, als die traditionellen Märkte ihre erste Chance nutzten, die militärischen Eskalationen des Wochenendes zu bewerten; Öl stieg auf 77 USD, und asiatische Aktien fielen um 1,4 %."},{"title_de":"Mehr als 9 Mrd. USD fliehen aus Bitcoin- und Ether-ETFs in vier Monaten"}]}',
  response: [Object],
  usage: [Object],
  finishReason: 'stop',
  [cause]: Error [AI_TypeValidationError]: Type validation failed: Value: {"news":[{"title_de":"Hyperliquids Token steigt, während der Iran-Schock am Wochenende kaum offene Märkte findet","ai_wrapup_text_de":"HYPE stieg um etwa 6 % als iranspezifische Spannungen die Märkte erschütterten, während Bitcoin und andere Risikoinvestments fielen und die meisten traditionellen Märkte geschlossen waren. Hyperliquid, eine on‑Chain dezentrale Perpetuals-Börse, erfasste das frühe Wochenendvolumen und wurde zu einer wichtigen Plattform für Händler, die ihr Risiko während der Schließung ausdrücken wollten. Das Handelsvolumen auf der Plattform erreichte fast ein einmonatiges Hoch von rund 200 Mrd. USD, und der Preis von HYPE sprang auf etwa 32 USD, da gebührenbasierte Einnahmen und Token‑Rückkäufe, die direkt mit dieser Aktivität gekoppelt sind, die Nachfrage ankurbelt. Der Token ist Jahr‑zu‑Datum um rund 25 % gestiegen, bleibt jedoch unter seinem Septemberspike von ca. 58 USD. Analysten zufolge könnten immer online, nicht‑Kustodial-Plattformen wie Hyperliquid zu ersten Reaktionsplattformen für geopolitische Risiken werden und Wochenend‑Krisen in einen strukturellen Rückenwind für Handelsvolumen, Gebühren‑Einnahmen und HYPE‑Token‑Nachfrage verwandeln.","id":"903b420b-713c-490a-b9fe-a1cdb2be5f22","description_de":"Als sich die Spannungen über iranspezifische Schlagzeilen in diesem Wochenende zuspitzten, stieg Hyperliquids HYPE‑Token um etwa 6 %, da Händler zur immer‑Online dezentralen Perpetuals-Plattform wechselten, um ihr Risiko auszudrücken, während viele traditionelle Märkte geschlossen waren. Bitcoin und andere Risikoinvestments fielen mit steigenden Iran‑Spannungen, während Öl und Gold im Rahmen eines breiteren Risiko‑Abstiegs anstiegen."},{"title_de":"Bitcoin und Kryptowährungen unter Druck, während Öl um 6 % steigt und globale Märkte das US‑Iran-Konfliktpreiszenario einbeziehen","ai_wrapup_text_de":"Bitcoin fiel früh am Montag auf 66.702 USD, ein Rückgang von 1,1 %, nach einer kurzen Sonntags‑Rallye, da die Märkte das US‑Iran-Konflikt und einen Ölpreisanstieg von 6 % preisgaben. Brent‑Crude stieg zu Beginn um bis zu 13 % und schloss bei rund 77,50 USD, während die Befürchtung, dass der Hohrizum‑Kanal effektiv verschlossen sei, die Stimmung gegenüber Risikoinvestments dämpfte. Ethereum, Solana und XRP schüttelten sich ebenfalls ab; Solana hatte in der Woche 8,1 % verloren und war mit der größten Verlustquote unter den Hauptakteuren. Einige Händler argumentieren, dass das Abwärtsrisiko begrenzt sei, weil die Welt weitgehend von iranischem Öl abgeknöpft sei; die Preisbewegungen hängen jedoch davon ab, ob der Hohrizum‑Kanal wieder öffnet und wie lange Trumps Ziele dauern werden.","id":"c341fa06-a8eb-4ed3-877d-25358151d98d","description_de":"Bitcoin sank auf 66.700 USD, als die traditionellen Märkte ihre erste Chance nutzten, die militärischen Eskalationen des Wochenendes zu bewerten; Öl stieg auf 77 USD, und asiatische Aktien fielen um 1,4 %."},{"title_de":"Mehr als 9 Mrd. USD fliehen aus Bitcoin- und Ether-ETFs in vier Monaten"}]}.
  Error message: [
    {
      "code": "invalid_type",
      "expected": "string",
      "received": "undefined",
      "path": [
        "news",
        2,
        "ai_wrapup_text_de"
      ],
      "message": "Required"
    },
    {
      "code": "invalid_type",
      "expected": "string",
      "received": "undefined",
      "path": [
        "news",
        2,
        "description_de"
      ],
      "message": "Required"
    },
    {
      "code": "invalid_type",
      "expected": "string",
      "received": "undefined",
      "path": [
        "news",
        2,
        "id"
      ],
      "message": "Required"
    }
  ]
      at async GET (app/api/test/route.ts:61:14)
    59 |        }
    60 |
  > 61 |        const res = await generateObject({
       |                    ^
    62 |                model: 'openai/gpt-oss-20b',
    63 |                messages: [
    64 |                        { {
    value: { news: [Array] },
    [cause]: Error [ZodError]: [
      {
        "code": "invalid_type",
        "expected": "string",
        "received": "undefined",
        "path": [
          "news",
          2,
          "ai_wrapup_text_de"
        ],
        "message": "Required"
      },
      {
        "code": "invalid_type",
        "expected": "string",
        "received": "undefined",
        "path": [
          "news",
          2,
          "description_de"
        ],
        "message": "Required"
      },
      {
        "code": "invalid_type",
        "expected": "string",
        "received": "undefined",
        "path": [
          "news",
          2,
          "id"
        ],
        "message": "Required"
      }
    ]
        at async GET (app/api/test/route.ts:61:14)
      59 |      }
      60 |
    > 61 |      const res = await generateObject({
         |                  ^
      62 |              model: 'openai/gpt-oss-20b',
      63 |              messages: [
      64 |                      { {
      issues: [ [Object], [Object], [Object] ],
      addIssue: [Function (anonymous)],
      addIssues: [Function (anonymous)]
    }
  }
}

I will try to upgrade to v6 and get back to you.

Hi there, thanks! Seems like my initial answer was hidden, can you see it?

Hey, I can see it. It was mistakingly flagged by our AI moderation. I’ll take a look at it now.

I just upgraded to v6 and used the generateText with the new syntax like this:

Code
const systemPrompt = `
		<context_gathering>
			Du bist ein professioneller Ăśbersetzer fĂĽr Krypto-News. 
			Du bekommst eine Liste von News-Artikeln und musst ALLE ĂĽbersetzen. 
			Die Anzahl der zurĂĽckgegebenen Artikel muss numerisch exakt der Eingabe entsprechen 
			(keine zusätzlichen Zeilenumbrüche, keine Auslassungen).
			Die Zielsprache ist Deutsch.
		</context_gathering>
	`

const res = await generateText({
   	model: 'openai/gpt-oss-20b',
   	system: systemPrompt,
   	prompt: `
   		Ăśbersetze alle Objekt-felder, auĂźer die 'id'.
   		Input data: ${JSON.stringify(input, null, 2)}
   	`,
   	output: Output.object({
   		schema: z.object({
   			news: z.array(
   				z.object({
   					ai_wrapup_text_de: z.string(),
   					description_de: z.string(),
   					id: z.string(),
   					title_de: z.string(),
   				})
   			),
   		}),
   	}),
   })

Still the same issue. I tried to only use groq with the providerOptions, same issue still.

I tried to change model to e.g. gpt-5-nano, the response then took almost 2 minutes but it responded successful.
Update: Got it down to 20-30 seconds with

providerOptions: {
	openai: {
		reasoningEffort: 'low',
	},
},

Then I tried to use google/gemini-2.5-flash-lite. First test 8sec and successful. Second one 30 seconds and also successful.

So to me this seems that the AI Gateway (or the providers) have some issues with latency, and especially some issues with gpt-* models.

1 Like

Hi @madebyfabian, thanks for sharing the code snippets. Could you also share the system prompt used in the above example?

Also, which region are you using this code in? For example, I’m sitting in the Netherlands right now.

Sure, I updated the Code example. I am in Nuremberg, Germany right now. So should be FRA.

I think changing everything to the gpt-oss-120b solves it for now. I could guarantee I checked it a few hours ago and it didn’t work with 120b either. So maybe the 20b will also be fixed in a few hours?

1 Like

Hi @madebyfabian, thanks for confirming the model behavior. I’m also testing some of the models you mentioned and for me:

  "sortedBySpeed": [
    {
      "model": "openai/gpt-oss-20b",
      "timeMs": 8553,
      "success": true
    },
    {
      "model": "google/gemini-2.5-flash-lite",
      "timeMs": 9887,
      "success": true
    },
    {
      "model": "anthropic/claude-haiku-4.5",
      "timeMs": 28163,
      "success": true
    },
    {
      "model": "gpt-5-nano",
      "timeMs": 72940,
      "success": true
    }
  ],

It maybe a transitional error or degraded performance of some sort. I’ll keep an eye out for similar posts and notify the AI Gateway team.

In the meantime, I hope you can switch out for a different model.

1 Like

Hi there!
What was the response from the team regarding the latency spikes?
Since we talked, I fully moved to gpt-oss-120b, and now everything mostly works fine. But sometimes (1-3x a day), I get extremely high latency (60-400+ seconds) and failing AI requests. So similar to the original issue, just not that often.

Hi @madebyfabian, it appeared to be a transitory issue back then but since you’re facing it again even with this model. I’ll ask the team again what could be wrong here.

Hi there, have you heard anything back? We still have those latency and failure rate issues. Noticing that it’s mostly at night time (EST). Or for like 2-3 hours on daytime until it gets back to regular. It’s causing a lot of issues in our pipelines, hoping it can get resolved otherwise we will have to migrate away from AI Gateway, which I would like to avoid. I am still running the gpt-oss-120b. For example the anthropic models work completely fine all the time in the last few days.

@madebyfabian thanks for sharing additional research about this. I’ll check with the team again about this model. I’m sorry this is taking longer. In the meantime, you can maybe use Anthropic as Model Fallbacks.