Jan 20
8
Google’s John Mueller responded to a question related to crawl budget around large scale web sites saying “crawling at a large scale is much more complicated.” This was when he proposed two ideas for improving crawl budget but John “both of those are too simplified” and there is more to it.
Link:
Google: Crawling At Large Scale Is Complicated; Don’t Oversimplify It