Dec 19
14
Google’s John Mueller said on Twitter that Google “never supported crawl-delay in robots.txt.” Bing’s Frédéric Dubut said on Twitter “Bingbot supports crawl-delay.” That is not to say Google does not let you slow the crawl, you can and here is how.
Go here to see the original:
Google Does Not Support Crawl Delay But Bing Does