Latency of AI Gateway in AISDK

AI SDK
1

What latency can we expect when routing requests via AI Gateway compared to native providers?
We use Gemini models via the Vertex provider and see between 300ms to 700ms additional latency when routing via gateway instead of direct vertex. Is this the expected performance?
We have inferenceRegion: { scope: 'zone', geoRegion: 'eu' }, configured in our setup.

2

Hi Flo,

I’d avoid treating 300–700ms as expected Gateway overhead until you confirm the Gateway request is hitting the same upstream path as your direct Vertex request.

For a narrow benchmark, I’d first pin the provider route so you’re comparing:

direct Vertex
vs
AI Gateway → Vertex

For example:

const result = await generateText({
  model: "google/gemini-...",
  prompt,
  providerOptions: {
    gateway: {
      only: ["vertex"],
    },
  },
})

console.log(JSON.stringify(result.providerMetadata?.gateway, null, 2))

If the metadata shows a different resolved provider, a fallback attempt, or more than one provider attempt, that could explain the added latency.

I’d also measure first-token latency separately from full completion time, because total duration can hide whether the delay is before the provider starts responding or during generation. For the comparison, keep these identical:

- same model ID / model variant
- same prompt and max output settings
- same Vercel Function region/runtime
- same streaming vs non-streaming mode
- same Vertex location / EU routing assumptions

The routing options are here if you want to pin or order providers during testing:

Can you share the redacted providerMetadata.gateway.routing output from one slow Gateway request?