[ZWeb] FYI: Readying the "Zope Download Center"

Guido van Rossum guido@python.org
Sun, 12 Jan 2003 09:02:15 -0500

> > Have you read my general mirroring proposal?  The help we need
> > right now would concern the architecture, design and
> > implementation of the system I propose there.  It shouldn't be
> > much code, since the essence of mirroring can be done using rsync
> > or ftpmirror.py, but we need tools to find downloadable bits, and
> > to collate download stats.
> Sure, though given the history of such efforts, I don't think we
> should wait for new software before fixing this problem.  We've
> talked for over two years about writing software to collect download
> stats, even before spreading download across multiple sites!

Hm.  For python.org, we use something called Webalizer, which scans
Apache-style log file.  Since Zope wrotes those too, I'm not sure what
the problem is for a single site.  Integrating the results back into

> The SF-based Download Center will be a step forward.


> Improving a Zope-based download, based on new software and a new
> mirror network, is also a good idea.


> > It would be best to coordinate with Sidnei, who's doing a
> > "downloadable product" for NZO.
> Actually I wrote the original one, so I have a rough idea what it does. 


>   One of the continuing problems: everyone says counters shouldn't be 
> done inside the ZODB (though getting a clear answer on this isn't easy).

It's true.  (Anecdote: during my brief stint at BeOpen in 2000, we
tried to use Zope for a commercial Python site.  The designers had
made a very attractive layout, and some Zope contractors had
implemented it all in Zope.  Unfortunately they had used a naive
ad-serving product that used ZODB counters, and nobody understood why
the site's Data.fs grew to unacceptable sizes until we called in Jim
Fulton who quickly diagnosed the problem.)

Three solutions come to mind:

- Use an auto-packing Berekely storage

- Use an undo-less separate storage for the counters

- Don't update the counters directly in ZODB; nightly, computer
  counters from log files and update them in ZODB

> I'll reply to your note from yesterday after this one...

Separate reply will follow.

--Guido van Rossum (home page: http://www.python.org/~guido/)