Also known as the robots exclusion standard or robots.txt, this is a standard used by websites that communicates directly to web crawlers, informing them which areas of the website should not be read. It is a plain text file that is uploaded to the root directory of the website and then linked in the html. Robots.txt provides instructions about the current page that can then be used by different search engines to categorize websites. Although the protocol uses the terms “allow” and “disallow” it is purely advisory and relies fully on the compliance of web robots.
You are here: / / Robots Exclusion Protocol