I noticed that OpenAI has been hitting my website and hitting it hard. It will hit pages with query params like ?page=1513\ which obviously does not exist. I know I can deny AI Bots in Vercel which I have done in the meantime to stop OpenAI from hitting the website hard. However, I was wondering if there’s a better way to at least let OpenAI and other AI bots know what pages are actually available so it’s not just randomly scraping my website over and over again with incorrect params and paths.
Here is the User Agent according to Vercel
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)