Oct 24
11
There is this interesting conversation on LinkedIn around a robots.txt serves a 503 for two months and the rest of the site is available. Gary Illyes from Google said that when other pages on the site are reachable and available, that makes a big difference, but when those other pages are not, then “you’re out of luck,” he wrote.
See the original post:
Google: Robots.txt Is Unreachable, Other Pages Reachability Matter