Before moving towards X-Robots-Tag, let’s have a small discussion on Robots. As webmasters, we already know about this term. It allows you to hide pages, folders, sub-domains or any content from a search engine spider.
Search for “edward snowden petition” to find the petition filed through the White House’s Petition site, and you’ll see something odd. The petition has no description, because the White House won’t let Google crawl the page. But it’s not a move against Snowden,..
Need to remove URLs from the index? For a small number of URLs, this can be a relatively straightforward process — when you are dealing with thousands or tens of thousands, it can be a lot more complex, especially if you feel a great deal of urgency about the matter. This is what I am Please visit Search Engine Land for the full article.