Damn Yahoo Slurp
It looks like they're ignoring the robots exclusion protocol now. As we speak there are several spiders looking for things like
Viewing User Profile:
http://www.woodmann.com/forum/member.php?u=xxxxx
(LLXX and JMI were just being probed actually.. how did it feel? :devil
Viewing Calendar:
http://www.woodmann.com/forum/calendar.php?do=displaymonth&c=1&month=2&year=2007
Viewing Attachment...
Creating Event...
etc.
Now if you look at the robots.txt file I wrote (hardly a trade secret) you'll see that member.php, calendar.php etc. are on the Disallow list.
http://woodmann.com/robots.txt
As you say, there has been a sudden increase in Slurp activity and those bastids are obviously cheating or something is broken.
We Disallowed Slurp and msnbot completely on the old server, to conserve bandwidth, but this new server can handle it. Still, it looks like Slurp's being a Bad Cracker and we may need to nop them out..