#ai-gateway

Ryu Hi Flo, I’d avoid treating 300–700ms as expected Gateway overhead until you confirm the Gateway request is hitting the same upstream path as your direct Vertex request. For a narrow benchmark, I’d first pin the provider route so you’re comparing: ``` direct Vertex vs AI Gateway → Vertex ``` For example: ``` const result = await generateText({ model: "google/gemini-...", prompt, providerOptions: { gateway: { only: ["vertex"], }, }, }) console.log(JSON.stringify(result.providerMetadata?.gateway, null, 2)) ``` If the metadata shows a different resolved provider, a fallback attempt, or more than one provider attempt, that could explain the added latency. I’d also measure first-token latency separately from full completion time, because total duration can hide whether the delay is before the provider starts responding or during generation. For the comparison, keep these identical: ``` - same model ID / model variant - same prompt and max output settings - same Vercel Function region/runtime - same streaming vs non-streaming mode - same Vertex location / EU routing assumptions ``` The routing options are here if you want to pin or order providers during testing: https://vercel.com/docs/ai-gateway/models-and-providers/provider-options Can you share the redacted `providerMetadata.gateway.routing` output from one slow Gateway request?

Justin Harr Ah, turns our it IS outputting an vector image, the `mediaType` (image/png) on the returned image result is just incorrect. I'll leave this up in case it's an unknown defect but easy enough for me to work around.

Ryu Hi octoo, Those generation IDs and the exact UTC window are the right kind of evidence for this. Since all 31 requests completed and the main difference is latency while the resolved provider was `baseten`, I’d treat this as a provider/routing/capacity question first, not a prompt or request-shape issue. For the next probe run, I’d capture the Gateway metadata alongside your latency numbers: ``` console.log(JSON.stringify(await result.providerMetadata, null, 2)) ``` The useful fields to compare are the resolved/final provider, generation ID, whether any fallback happened, TTFT, total latency, input/output tokens, and timestamp. If this route has more than one usable provider, you could also do a short A/B probe with provider ordering pinned, using the provider slugs from the model page: ``` providerOptions: { gateway: { order: ["<preferred-provider>", "baseten"], }, } ``` If the same prompts stay fast when routed elsewhere but slow only on `baseten`, that makes the report much easier to distinguish from prompt size or structured-output parsing overhead. If every provider is slow during the same window, then it points more toward Gateway-level routing/queueing or the model route overall. For escalation, I’d include a compact table with generation ID, provider, UTC time, TTFT, total latency, token counts, and whether it was from the affected or baseline run.

Amy Egan I wasn't able to repro, but maybe I'm looking in the wrong place. Can you show me where you see standard tier pricing for these requests? AI Gateway supports OpenAI tiers through the `service_tier` param. Looks like you're using it right. https://vercel.com/docs/ai-gateway/capabilities/service-tiers

Mo091336 4719 Hi Amy, Fortunately, the issue with the GPT-5.5 Flex and Priority tiers was resolved just a few days after I wrote my post, before you had a chance to reply. It appears that Vercel simply had not updated its billing system to reflect the new pricing structure, since the provider was returning responses with the correct service tier. However, the same issue reappeared with the release of the GPT-5.6 series, apparently for exactly the same reason. This time, there also seems to be an issue with the updated caching system. OpenAI has introduced both implicit and explicit prompt caching, but Vercel still appears to be using the previous configuration. As a result, implicit caching is being applied by default, with input charged at 1.25 times the standard rate and cache breakpoints not being respected. Could you please take a look at this? P.S. The same behavior occurs with both the Chat Completions API and the Responses API.

Mo091336 4719 Update: Now It works. Thank you Amy.

briascoi Additional evidence: I completed the supported upgrade flow from Settings → Drains. The checkout reached the final confirmation, but the server rejected the upgrade with: “Only pro trial subscriptions can be modified.” At the same time, the team API reports plan=pro and status=trialing, while Billing already shows a paid USD 20 Pro invoice dated August 7. This confirms an inconsistent subscription/entitlement state on Vercel’s side. Please reconcile the billing subscription and the team entitlement. I will not retry the payment because USD 20 has already been charged.

Amy Egan It looks like your team is on the Pro plan now. AI Gateway credits are billed separately, so you should be able to purchase credits to upgrade from the free tier from the `AI Gateway` tab in your team dashboard. You can also configure an auto-reload amount and spend limits. Please let me know if you still have any trouble with this. I'd be happy to help you get it working for your project

Weilue Luo /v1/models working only means the gateway isreachable. since /v1/credits is also 403, vercel probably need time and x-vercel-id to dig deeper