Feb 25
26
Last April 2024, Gary Illyes from Google said he was on a mission to make web crawling more efficient, he wanted to “figure out how to crawl even less, and have fewer bytes on wire.” Gary updated us saying “there’s lots more in works” for this but implies some improvements have already been made.
Read the original post:
Google: There Is A Lot More Work To Do To Make Crawling More Efficient