We observed a significant latency increase for AI Gateway requests to:
Model: openai/gpt-oss-120b
Resolved provider: baseten
Approximate affected window: 2026-06-29 15:00-15:05 UTC
This was during a repeatable internal probe/validation run using the same model path and similar structured-output prompts as prior runs. The requests completed successfully, but latency
increased across the full probe set.
Affected run summary:
- Requests: 31
- Successful responses: 31
- Failed/errored responses: 0
- Total tokens: about 64.7k
- Input tokens: about 56.9k
- Output tokens: about 7.8k
- Median latency: about 6.95s
- Average latency: about 6.88s
- p95 latency: about 10.97s
For comparison, recent prior runs on the same route were typically:
- Average latency: 1.0-1.3s
- Median latency: 0.9-1.1s
- p95 latency: 1.5-2.4s
Sample slow generation IDs from the affected window, with two faster prior same-route samples for comparable validation prompts:
Affected generation Affected latency Prior fast samples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gen_01KW9YDE2B45WPG0MMKBDB4MYY about 19.1s gen_01KW5471SGENBV90ZXBJSDMYFX about 2.1s; gen_01KW58JFB1RZ1HWVR2Y24B98FP about 2.2s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YFR6MCPZRV4N8481TY642 about 16.8s gen_01KW8JTMDDC5GB9Z2R39DN63GK about 1.5s; gen_01KW7M99N4TX9E0BCBXNKHVMED about 1.5s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YGSG7QAGJ75EYEFAVRWGV about 11.0s gen_01KW58JZE2S7VNCQGN3AD2J3KS about 1.6s; gen_01KW7M9EDHE6XH1DNKAADCAJPP about 1.7s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YKG2RFPV8H82AT0XMSBPC about 10.0s gen_01KW572V409E6ERNTBDXG2T24W about 0.9s; gen_01KW58KEQP23201ZBQD430APK1 about 1.0s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YEWCVFT5BBBDRQRNTESDS about 9.2s gen_01KW5478W9MSZFG6AEK293BMYC about 0.7s; gen_01KW7M96BZT1DAN9NY9MC859FJ about 0.9s
Conformance-style probe requests immediately before the full validation were also slower than usual:
Affected probe generation Affected latency Prior fast samples
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
gen_01KW9YAXCW87NQX6KRTCSSSSGG about 2.2s gen_01KW7M6D9M5AAVX2MYJ832V84C about 0.7s; gen_01KW58GFS0QCK8NCF605KY133T about 0.9s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YAZFH3HFX17V80NK0HM5E about 2.8s gen_01KW544VS93S8R7C5JH3613VKE about 0.4s; gen_01KW58GGHT66ACF59WT3HPH496 about 0.4s
──────────────────────────────── ────────────────── ──────────────────────────────────────────────────────────────────────────────────────
gen_01KW9YB27GM3WAMXCB9NZQ3949 about 3.2s gen_01KW544W8AVD6E0B8Z54TQQ2SW about 0.5s; gen_01KW7M6EHYH6KGDXZ74SQGEPQ6 about 0.5s
Can smbd help whether there was elevated latency, capacity pressure, routing behavior, or backend degradation for openai/gpt-oss-120b via Baseten around 2026-06-29 15:00-15:05 UTC?