robots.txt checker This tool validates /robots.txt files according to the robots exclusion de-facto standard. Checks are done considering the original 1994 document A Standard for Robot Exclusion, the 1997 Internet Draft specification A Method for Web Robots Control and nonstandard extensions that have emerged over the years. The robots.txt validator checks the syntax and structure of the document