MG Marketing » spiders http://mg-mk.com/en Online Marketing Agency Mon, 20 Apr 2026 18:01:12 +0000 http://wordpress.org/?v=2.9.1 en hourly 1 OpenAI Updates Its ChatGPT Crawler: OAI-SearchBot http://mg-mk.com/en/openai-updates-its-chatgpt-crawler-oai-searchbot/ http://mg-mk.com/en/openai-updates-its-chatgpt-crawler-oai-searchbot/#comments Tue, 09 Dec 2025 12:51:00 +0000 admin http://mg-mk.com/en/openai-updates-its-chatgpt-crawler-oai-searchbot/ OpenAI has updated its spider/crawler it uses for ChatGPT, OAI-SearchBot.

See original here:
OpenAI Updates Its ChatGPT Crawler: OAI-SearchBot

]]> http://mg-mk.com/en/openai-updates-its-chatgpt-crawler-oai-searchbot/feed/ 0 OpenAI Scales Up Crawling & Bots For The Holidays http://mg-mk.com/en/openai-scales-up-crawling-bots-for-the-holidays/ http://mg-mk.com/en/openai-scales-up-crawling-bots-for-the-holidays/#comments Fri, 28 Nov 2025 12:51:00 +0000 admin http://mg-mk.com/en/openai-scales-up-crawling-bots-for-the-holidays/ OpenAI is reportedly scaling up its crawling infrastructure for the holiday shopping season. The folks at Merj noticed OpenAI adding a lot of new IP ranges for its bots and crawlers.

See the original post here:
OpenAI Scales Up Crawling & Bots For The Holidays

]]> http://mg-mk.com/en/openai-scales-up-crawling-bots-for-the-holidays/feed/ 0 Cloudflare To Block AI Crawlers By Default & Pay Per Crawl Model http://mg-mk.com/en/cloudflare-to-block-ai-crawlers-by-default-pay-per-crawl-model/ http://mg-mk.com/en/cloudflare-to-block-ai-crawlers-by-default-pay-per-crawl-model/#comments Tue, 01 Jul 2025 11:31:00 +0000 admin http://mg-mk.com/en/cloudflare-to-block-ai-crawlers-by-default-pay-per-crawl-model/ Cloudflare, a company that powers about 20% of all web pages on the web, has announced it is now blocking AI crawlers by default. Plus, it is offering a new model to allow AI services to pay content creators to crawl their content, named Pay Per Crawl.

Link:
Cloudflare To Block AI Crawlers By Default & Pay Per Crawl Model

]]> http://mg-mk.com/en/cloudflare-to-block-ai-crawlers-by-default-pay-per-crawl-model/feed/ 0 Bing Crawl Consumption Not Showing Return On Investment http://mg-mk.com/en/bing-crawl-consumption-not-showing-return-on-investment/ http://mg-mk.com/en/bing-crawl-consumption-not-showing-return-on-investment/#comments Thu, 31 May 2018 19:24:43 +0000 admin http://mg-mk.com/en/bing-crawl-consumption-not-showing-return-on-investment/ Joost de Valk, the founder of Yoast, posted some interesting data on Twitter yesterday around crawlers and how much they consume of their site, how active they are and if there is any return on investment. The big one is that Bing crawled ~84…

See original here:
Bing Crawl Consumption Not Showing Return On Investment

]]> http://mg-mk.com/en/bing-crawl-consumption-not-showing-return-on-investment/feed/ 0 Is Your Local Pricing Strategy Blocking Search Engine Spiders? http://mg-mk.com/en/is-your-local-pricing-strategy-blocking-search-engine-spiders/ http://mg-mk.com/en/is-your-local-pricing-strategy-blocking-search-engine-spiders/#comments Tue, 12 Feb 2013 15:44:36 +0000 admin http://mg-mk.com/en/is-your-local-pricing-strategy-blocking-search-engine-spiders/ eMarketer’s recent report on Global e-commerce growth showed online sales globally exceeded $1 trillion in 2012. They further indicate that global e-commerce will grow by an additional 19% in 2013, with the Asia-Pacific region surpassing North America in online sales. This reemphasizes the…

See the rest here:
Is Your Local Pricing Strategy Blocking Search Engine Spiders?

]]> http://mg-mk.com/en/is-your-local-pricing-strategy-blocking-search-engine-spiders/feed/ 0 SpiderDuck: The Realtime Twitter Spider http://mg-mk.com/en/spiderduck-the-realtime-twitter-spider/ http://mg-mk.com/en/spiderduck-the-realtime-twitter-spider/#comments Fri, 18 Nov 2011 13:38:54 +0000 admin http://mg-mk.com/en/spiderduck-the-realtime-twitter-spider/ Twitter has unveiled their URL fetcher, which they named SpiderDuck…

See the rest here:
SpiderDuck: The Realtime Twitter Spider

]]> http://mg-mk.com/en/spiderduck-the-realtime-twitter-spider/feed/ 0 Google: Remove The Robots.txt File Completely http://mg-mk.com/en/google-remove-the-robots-txt-file-completely/ http://mg-mk.com/en/google-remove-the-robots-txt-file-completely/#comments Thu, 06 Jan 2011 20:54:21 +0000 admin http://mg-mk.com/en/google-remove-the-robots-txt-file-completely/ Believe it or not, I am not a huge fan of placing robots.txt files on sites unless you want to specifically block content and sections from Google or other search engines. It just always felt redundant to tell a search engine they can crawl your site when they will do so unless you tell them not to.

Read the original post:
Google: Remove The Robots.txt File Completely

]]> http://mg-mk.com/en/google-remove-the-robots-txt-file-completely/feed/ 0