Dec 20
15
Google’s John Mueller said he would advice that if “that’s something where if you see that these pages are important enough that people are linking to them then I would try to avoid blocking them by robot’s txt.” In short, if you have important or popular pages with a lot of links to them, make sure Google can access the page.
Read more:
Google: Avoid Blocking Pages That Are Important Enough To Have Links To Them