Community Matters

It was just a couple of weeks ago that I posted about the number of undesirable bots crawling my site. Over the last few days suddenly the number of genuine bots has just exploded. I’m back to more than two thirds of the requests coming from bots (actually closer to 80% at this point). I’m wondering if this is normal - and if I am spending too much time watching my log files :-)

I can identify at least eight different blog aggregators scanning my feeds - Technorati takes the price for most impatient with an access every ten minutes. I see a similar number of search engines, at least two of them getting thoroughly confused by my old Blosxom sites (that I mostly leave around so that they are available for people who have linked to them). As a result these bots are searching very odd URLs that are valid but redundant. I wonder if this helps or hurts my page ranking…

738 requests from Google in a day seems just a wee bit overkill. I am not posting THAT much. And I have a sitemap that theoretically tells Google which URLs to crawl and how frequently they are likely to change. Heck, it’s a protocol that they invented!

People are writing a lot about the fact that a large part of the internet traffic is spam. I am beginning to wonder how much of the web traffic is actual end users compared to bots crawling.

Thanks for visiting!
I hope this was helpful - if not, please leave a comment and let me know why! Were you searching for something else? Did I miss an important aspect?