MG Marketing » seo: blocking spiders http://mg-mk.com/en Online Marketing Agency Tue, 21 Apr 2026 15:16:00 +0000 http://wordpress.org/?v=2.9.1 en hourly 1 Apple Confirms Their Web Crawler: Applebot http://mg-mk.com/en/apple-confirms-their-web-crawler-applebot/ http://mg-mk.com/en/apple-confirms-their-web-crawler-applebot/#comments Wed, 06 May 2015 13:23:09 +0000 admin http://mg-mk.com/en/apple-confirms-their-web-crawler-applebot/ Apple finally confirms they have a web crawler, primarily used for Siri and Spotlight Suggestions. The post Apple Confirms Their Web Crawler: Applebot appeared first on Search Engine Land.

Link:
Apple Confirms Their Web Crawler: Applebot

]]> http://mg-mk.com/en/apple-confirms-their-web-crawler-applebot/feed/ 0 How Search Engines Process Links http://mg-mk.com/en/how-search-engines-process-links/ http://mg-mk.com/en/how-search-engines-process-links/#comments Mon, 13 Apr 2015 13:41:29 +0000 admin http://mg-mk.com/en/how-search-engines-process-links/ Ever wondered how search engines crawl, analyze, index, and rank pages? Columnist Jenny Halasz has created a helpful primer on the link graph to answer these questions. The post How Search Engines Process Links appeared first on Search Engine Land

Continue reading here:
How Search Engines Process Links

]]> http://mg-mk.com/en/how-search-engines-process-links/feed/ 0 Google Now Supports Crawling & Indexing Locale-Adaptive Web Pages http://mg-mk.com/en/google-now-supports-crawling-indexing-locale-adaptive-web-pages/ http://mg-mk.com/en/google-now-supports-crawling-indexing-locale-adaptive-web-pages/#comments Wed, 28 Jan 2015 13:57:15 +0000 admin http://mg-mk.com/en/google-now-supports-crawling-indexing-locale-adaptive-web-pages/ Google adds support for web pages that dynamically change their content based on IP origin or language settings. The post Google Now Supports Crawling & Indexing Locale-Adaptive Web Pages appeared first on Search Engine Land.

The rest is here:
Google Now Supports Crawling & Indexing Locale-Adaptive Web Pages

]]> http://mg-mk.com/en/google-now-supports-crawling-indexing-locale-adaptive-web-pages/feed/ 0 Google Mobile Search Descriptions Now Say If You’re Blocking Google http://mg-mk.com/en/google-mobile-search-descriptions-now-say-if-you%e2%80%99re-blocking-google/ http://mg-mk.com/en/google-mobile-search-descriptions-now-say-if-you%e2%80%99re-blocking-google/#comments Tue, 27 Jan 2015 13:50:50 +0000 admin http://mg-mk.com/en/google-mobile-search-descriptions-now-say-if-you%e2%80%99re-blocking-google/ Yet another reason not to block JavaScript, CSS, etc from Google.

See the original post here:
Google Mobile Search Descriptions Now Say If You’re Blocking Google

]]> http://mg-mk.com/en/google-mobile-search-descriptions-now-say-if-you%e2%80%99re-blocking-google/feed/ 0 Google Orders Terminator Robots Not To Kill Founders Brin & Page http://mg-mk.com/en/google-orders-terminator-robots-not-to-kill-founders-brin-page/ http://mg-mk.com/en/google-orders-terminator-robots-not-to-kill-founders-brin-page/#comments Thu, 03 Jul 2014 17:09:22 +0000 admin http://mg-mk.com/en/google-orders-terminator-robots-not-to-kill-founders-brin-page/ How do you stop the unstoppable killer Terminators if you’re not Sarah Connor? Google does it with a simple text file. People have noted today that Google has a special “robot.txt” file that pokes fun at stopping the Terminators.

Continued here:
Google Orders Terminator Robots Not To Kill Founders Brin & Page

]]> http://mg-mk.com/en/google-orders-terminator-robots-not-to-kill-founders-brin-page/feed/ 0 Robots.txt Celebrates 20 Years Of Blocking Search Engines http://mg-mk.com/en/robots-txt-celebrates-20-years-of-blocking-search-engines/ http://mg-mk.com/en/robots-txt-celebrates-20-years-of-blocking-search-engines/#comments Mon, 30 Jun 2014 13:14:49 +0000 admin http://mg-mk.com/en/robots-txt-celebrates-20-years-of-blocking-search-engines/ Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites too hard.

See original here:
Robots.txt Celebrates 20 Years Of Blocking Search Engines

]]> http://mg-mk.com/en/robots-txt-celebrates-20-years-of-blocking-search-engines/feed/ 0 Google Fixing Reverse DNS GoogleBot Verification http://mg-mk.com/en/google-fixing-reverse-dns-googlebot-verification/ http://mg-mk.com/en/google-fixing-reverse-dns-googlebot-verification/#comments Thu, 03 Apr 2014 16:09:41 +0000 admin http://mg-mk.com/en/google-fixing-reverse-dns-googlebot-verification/ Google has confirmed that there are some GoogleBot useragent spiders not properly passing verification protocol. Savvy webmasters noticed that GoogleBot over the .249.70.0 /24 IP range was not returning the proper reverse DNS verification details. The response given was “no such host is..

Here is the original post:
Google Fixing Reverse DNS GoogleBot Verification

]]> http://mg-mk.com/en/google-fixing-reverse-dns-googlebot-verification/feed/ 0 Google’s Matt Cutts: NoFollow Attributes On Internal Links Don’t Hurt But Generally Don’t Do It http://mg-mk.com/en/google%e2%80%99s-matt-cutts-nofollow-attributes-on-internal-links-don%e2%80%99t-hurt-but-generally-don%e2%80%99t-do-it/ http://mg-mk.com/en/google%e2%80%99s-matt-cutts-nofollow-attributes-on-internal-links-don%e2%80%99t-hurt-but-generally-don%e2%80%99t-do-it/#comments Mon, 30 Sep 2013 18:45:29 +0000 admin http://mg-mk.com/en/google%e2%80%99s-matt-cutts-nofollow-attributes-on-internal-links-don%e2%80%99t-hurt-but-generally-don%e2%80%99t-do-it/ In a video, Google’s head of search spam Matt Cutts published today an answer to the question, “Should I use rel=”nofollow” on internal links to a login page?” Matt Cutts basically said you shouldn’t, but said it won’t hurt you if you did. Matt said,… Please visit Search Engine Land for the full article.

See more here:
Google’s Matt Cutts: NoFollow Attributes On Internal Links Don’t Hurt But Generally Don’t Do It

]]> http://mg-mk.com/en/google%e2%80%99s-matt-cutts-nofollow-attributes-on-internal-links-don%e2%80%99t-hurt-but-generally-don%e2%80%99t-do-it/feed/ 0 Snowden Petition Blocked From Google? Like All Petitions, It Won’t Be When It Gets Enough Signatures http://mg-mk.com/en/snowden-petition-blocked-from-google-like-all-petitions-it-won%e2%80%99t-be-when-it-gets-enough-signatures/ http://mg-mk.com/en/snowden-petition-blocked-from-google-like-all-petitions-it-won%e2%80%99t-be-when-it-gets-enough-signatures/#comments Fri, 28 Jun 2013 13:43:43 +0000 admin http://mg-mk.com/en/snowden-petition-blocked-from-google-like-all-petitions-it-won%e2%80%99t-be-when-it-gets-enough-signatures/ Search for “edward snowden petition” to find the petition filed through the White House’s Petition site, and you’ll see something odd. The petition has no description, because the White House won’t let Google crawl the page. But it’s not a move against Snowden,..

Read more:
Snowden Petition Blocked From Google? Like All Petitions, It Won’t Be When It Gets Enough Signatures

]]> http://mg-mk.com/en/snowden-petition-blocked-from-google-like-all-petitions-it-won%e2%80%99t-be-when-it-gets-enough-signatures/feed/ 0 Twitter Opens Up To More Crawling, But Do Search Engines Want Its Search Results In Theirs? http://mg-mk.com/en/twitter-opens-up-to-more-crawling-but-do-search-engines-want-its-search-results-in-theirs/ http://mg-mk.com/en/twitter-opens-up-to-more-crawling-but-do-search-engines-want-its-search-results-in-theirs/#comments Thu, 27 Sep 2012 14:05:22 +0000 admin http://mg-mk.com/en/twitter-opens-up-to-more-crawling-but-do-search-engines-want-its-search-results-in-theirs/ Twitter recently updated its robots.txt file and, though the change opens up millions of pages to being crawled, there’s no guarantee that the main search engines want what Twitter is offering. The Sociable seems to have been the first to notice Twitter’s robots.txt changes, which now… Please visit Search Engine Land for the full article

Visit link:
Twitter Opens Up To More Crawling, But Do Search Engines Want Its Search Results In Theirs?

]]> http://mg-mk.com/en/twitter-opens-up-to-more-crawling-but-do-search-engines-want-its-search-results-in-theirs/feed/ 0