Hi Flo,
I’d avoid treating 300–700ms as expected Gateway overhead until you confirm the Gateway request is hitting the same upstream path as your direct Vertex request.
For a narrow benchmark, I’d first pin the provider route so you’re comparing:
1direct Vertex2vs3AI Gateway → Vertex4
For example:
1const result = await generateText({2 model: "google/gemini-...",3 prompt,4 providerOptions: {5 gateway: {6 only: ["vertex"],7 },8 },9})1011console.log(JSON.stringify(result.providerMetadata?.gateway, null, 2))12
If the metadata shows a different resolved provider, a fallback attempt, or more than one provider attempt, that could explain the added latency.
I’d also measure first-token latency separately from full completion time, because total duration can hide whether the delay is before the provider starts responding or during generation. For the comparison, keep these identical:
1- same model ID / model variant2- same prompt and max output settings3- same Vercel Function region/runtime4- same streaming vs non-streaming mode5- same Vertex location / EU routing assumptions6
The routing options are here if you want to pin or order providers during testing:
https://vercel.com/docs/ai-gateway/models-and-providers/provider-options
Can you share the redacted providerMetadata.gateway.routing output from one slow Gateway request?