Nov 10
26
Webmasters and SEOs have new reading material for this weekend. Google has published a very comprehensive and detailed Robots.txt Specifications, Robots meta tag and X-Robots-Tag HTTP header specifications and how to control crawling and indexing by GoogleBot. There are two…
Read more:
Google Published Detailed Specifications on Robots.txt