[ZWeb] Zope.org currently unusable
jens at dataflake.org
Thu Mar 10 09:39:43 EST 2005
On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:
> I need to read up on the robots.txt spec. Excellent Mark, thanks.
That piece is not part of the spec. Just like the wildcards that Google
claims they use (and I still don't believe that works as advertised).
This is the spec:
Here is a robots.txt validator:
Here's a funny one: Some collected all the reckless/useless user agents
This one explains Slurp-specific extensions:
More information about the Zope-web