[ZWeb] Zope.org currently unusable

Jens Vagelpohl jens at dataflake.org
Thu Mar 10 09:39:43 EST 2005

On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:

> I need to read up on the robots.txt spec.  Excellent Mark, thanks.
> Andrew

That piece is not part of the spec. Just like the wildcards that Google 
claims they use (and I still don't believe that works as advertised). 
This is the spec:


Here is a robots.txt validator:


Here's a funny one: Some collected all the reckless/useless user agents 
for exclusion:


This one explains Slurp-specific extensions:



More information about the Zope-web mailing list