In the early days of the internet, robots went by many names: spiders, crawlers, worms, WebAnts, web crawlers. Most of the time, they were built with good intentions.

A web crawler (also known as a web spider or web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. This process is called Web crawling or ...

Use robots.txt to block crawlers from "action URLs." This prevents wasted server resources from useless crawler hits. It's an age-old best practice that remains relevant today.

If all crawlers are to be blocked, the robots.txt looks like this: User-agent: * Disallow: / Information on robots.txt can be found at Open AI and at Google.