Jul 19
3
Google announced yesterday as part of its efforts to standardizing the robots exclusion protocol that it is open sourcing its robots.txt parser. That means how GoogleBot reads and listens to robots.txt files will be available for any crawler or coder to look at or use.
More here:
Google Shares Its Robots.txt Parser Code With Open Source World