Search crawlers are becoming more and more effective every day in discovering new content pages on websites.