Jul 24
2
Google’s Gary Illyes highlights robots.txt file’s error tolerance and unexpected features as it marks 30 years of aiding web crawling and SEO.
More here:
Robots.txt Turns 30: Google Highlights Hidden Strengths via @sejournal, @MattGSouthern