[ZWeb] Zope.org currently unusable
jens at dataflake.org
Thu Mar 10 04:33:28 EST 2005
On Mar 10, 2005, at 2:18, Andrew Sawyers wrote:
> It's a little of both; there's a group of people working on this - we
> to have something real soon now :) as a fix. Jens, could do you have
> time to check the zope.org robots.txt? A lot of the problems I've seen
> recently were due to several robots spidering zope.org at a time. I'm
> working on additional hardware and we should see more traction on the
> project sooner then later.
I don't believe all that much in robots.txt. The nasty bots completely
ignore it, anyway. The only way to deal with them is to block them with
What's currently there looks odd:
# Ask Google to skip search queries and the like.
Looking at the spec the case sensitivity of the User-agent value is
only "recommended", but you could shorten that into the following,
because multiple User-agent values are allowed per rule set:
Otherwise there really isn't much in there, and from seeing googlebots
myself often enough I have my doubts whether the line "Disallow: /*?"
works at all.
More information about the Zope-web