[ZWeb] google results duplication

Shane Hathaway shane@digicool.com
Fri, 11 May 2001 22:17:42 -0400

Karl Anderson wrote:
> Searching zope.org with google gets lots of duplicate results due to
> multiple views of many objects that contain the normal view content as
> well.  This makes google less useful for searching.
> Wiki pages are especially bad for this - the edit, history, backlinks,
> and advanced action views for each page end up in the google listing.
> Most of the other types have a prettyprinting page that shows up.
> Is there an easy way to exclude these?  Will robots.txt work for page
> suffixes, or just subdirectories?

ZWiki just needs to be fixed to not generate infinite URLs based on
acquisition.  If you browse around a ZWiki long enough, you'll find that
path elements in the URL get repeated.  Or you can just browse the
Zope.org logs to find googlebot and its excessive URLs.