[ZWeb] Zope.org currently unusable
mark at zopemag.com
Thu Mar 10 10:38:31 EST 2005
You are correct that the crawl-delay parameter is not part of the spec.
Their are plenty of examples where specs don't keep up with the times
and their is no harm done using that tag.
Worrying about a robots.txt file is a bit over the top :-)
Thanks for the link to the reckless/useless user agents page.
On Mar 10, 2005, at 3:39 PM, Jens Vagelpohl wrote:
> On Mar 10, 2005, at 15:27, Andrew Sawyers wrote:
>> I need to read up on the robots.txt spec. Excellent Mark, thanks.
> That piece is not part of the spec. Just like the wildcards that
> Google claims they use (and I still don't believe that works as
> advertised). This is the spec:
> Here is a robots.txt validator:
> Here's a funny one: Some collected all the reckless/useless user
> agents for exclusion:
> This one explains Slurp-specific extensions:
> Zope-web maillist - Zope-web at zope.org
More information about the Zope-web