Before moving towards X-Robots-Tag, let’s have a small discussion on Robots. As webmasters, we already know about this term. It allows you to hide pages, folders, sub-domains or any content from a search engine spider.