Current behavior:
When AI tools such as OpenAI’s ChatGPT (with browsing enabled) attempt to access my Vercel-hosted site https://pattern-paradise.shop, the request returns a 500 Internal Server Error. This includes both dynamic routes and even static assets like images.
Expected behavior:
The site should return a 200 OK response with the expected content, as it does for browsers, curl, and other tools. Static assets and public pages should be accessible to all visitors, including bots.
Reproduction Steps
Deploy a public Next.js site to Vercel.
Confirm that public pages and assets (e.g., .jpeg files in /public) work in a browser and via curl.
Attempt to access the same URL from ChatGPT (or similar AI agent) using its browsing tool.
The request results in a 500 Internal Server Error, with no log recorded in the Vercel dashboard.
Example failing URL for AI tools (but works in browser and curl):
Project Information
Framework: Next.js
Hosting platform: Vercel
Analytics: @vercel/analytics
robots.txt: Allows all bots (User-agent: *, with sitemap included)
Issue affects: All routes and assets, including static files
Additional Details
There are no log entries for these failed requests in Vercel’s logs (suggesting the error may occur before the request reaches the app).
Requests from browsers and tools like curl succeed — even using curl -A “ChatGPT”.
Only requests from AI agents (like ChatGPT browsing) trigger the 500 error.
Request for Help
Is Vercel blocking or rate-limiting headless/AI traffic at the edge or CDN layer?
Is there a recommended way to allow access from bots like ChatGPT?
Could this be a region-specific or user-agent-specific issue?
Any known workarounds or headers required for AI access?
Let me know what else I can provide to help reproduce or diagnose the issue. I’d appreciate any insight or advice on how to resolve this.
It sounds like these requests might be getting caught by our firewall. If you go to your project > Firewall at the top of the page, do you see requests getting blocked or challenged? The observability tab might also have some helpful details about where the request is getting stuck.
Additionally, where are your agents running and what kinds of headers are they using to identify themselves? It could be the case that requests are getting mistakenly filtered due to IP range abuse or something similar.
We haven’t seen any logs or app-level errors, so it does sound like a firewall-level issue. I just checked the Firewall tab and will monitor for challenged or blocked requests going forward.
Regarding the agent:
The request is coming from OpenAI’s ChatGPT, using their Browsing tool.
I don’t control the exact IP or headers, but based on curl tests, it likely uses a User-Agent similar to:
The request doesn’t include cookies, and may not include typical headers like Accept-Language or Referer.
For my platform, it would be more than ideal if AI agents could access and parse public data properly, not just for SEO, but also for integration into AI-driven search and discovery tools. Ensuring this access is reliable is a big part of my long-term strategy.
Is there a way to allow traffic from AI agents like ChatGPT or whitelist requests with known safe headers/IPs?
Hi Joyce, thanks for the details. To address the initial question - no, we’re not blocking AI or automated traffic unless it’s deemed to be malicious based on some other characteristic.
We maintain a list of known “good” bots here: Bot Protection. This doesn’t mean we necessarily block bots that aren’t on this lists, these are just the ones that we want to affirmatively allow.
I had a quick look at our firewall logs and didn’t see any blocked requests that looked like they came from OpenAI. Can you say more about how you’re initiating the request from your agent? For instance, is this a tool call in a chat, some kind of automation, etc.
