Jan 23
16
Top Story In 2017, Google’s Gary Illyes explained what crawl budget is, how crawl rate limits work, what crawl demand is and what factors impact a site’s crawl budget. As Illyes explained, most sites don’t need to worry about crawl budget. “Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-generate pages based on URL parameters,” Illyes said
See the article here:
This day in search marketing history: January 16