Made this reference going over the syntax of robots.txt, some examples and a list of all the major relevant bots that can be regulated.

robots.txt is a standard file at the root of websites to provide bot policies for search engines and AI models. It has its own unique syntax with just 4 keywords/commands.

https://stephcraft.net/docs/robots.txt