DreamHost is supposedly touted as one of the Best web hosting providers by some of it's users[including a few people I know personally of]. But it now seems that it is requiring users to edit the .htaccess file to block GoogleBot from indexing their pages. Are you wondering what the GoogleBot is all about - well it is the Google web crawler that indexes web pages for them to appear on a Google search. It seems that the explanation given by DreamHost is that GoogleBot is a "high memory usage and load on the server" and blocking it seems only the way out. The company has already dispatched off email's to most of it's customers, asking them to make their content unsearchable for spiders and crawlers. I guess the entire move by DreamHost is rather so STUPID!! Instead of them to upgrade its systems to make them more powerful, stable and able to support the crawl process of Googlebot - they ask you shut it down completely.
This will really be a Nightmare for many of it's customers since most visitors to a site come from Google searches and if you shut the GoogleBot out from indexing your website - your pages simply don't appear on Google search results anymore and you will see a dramatic drop in visitor numbers. If you take this site for example, a large % of visitors drop in from Google searches.
Below is an exact copy of the mail I got it forwarded.
This email is to inform you that a a few of your sites were getting hammered by Google bot. This was causing a heavy load on the webserver, and in turn affecting other customers on your shared server. In order to maintain stability on the webserver, I was forced to block Google bot via the .htaccess file. You also want to consider making your files be unsearchable by robots and crawlers, as that usually contributes to number of hits. If they hit a dynamic file, like php, it can cause high memory usage and consequently high load.
So is it really worth the dollars you spend on paid hosting. Well I guess NO. As for me I will stick for a long time with Blogger on the Free google platform for this blog :)
