[ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

[Chris Bainbridge]

2009/5/26 Pedro Ferreira <jose.pedro.ferreira at cern.ch>:

> In any case, it's not such a surprising number, since we have ~73141
> event objects and ~344484 contribution objects, plus ~492016  resource
> objects, and then each one of these may contain authors, and fore sure
> some associated objects that store different bits of info...

I had a similar problem packing a large 15GB database. This may sound
obvious, and is totally application specific, but you might like to
look into whether it really is necessary to have so many individual
ZODB objects. I found that often I would have a class that inherited
from one of the Persistent classes that was part of a larger
Persistent class, resulting in an explosion of Persistent objects. It
was possible to make many of them non-Persistent, so that they only
get serialised and stored in the database as part of whatever larger
Persistent object that they're part of. Another thing to watch out for
is dangling pointers - given the dynamic nature of Python, it's very
easy to leave a pointer hanging around to an old Persistent object
that you don't really need to store in the database, which in turn
points to more objects, etc. Any object that can be reached in a
traversal of the object graph will be kept in the database forever.
The lack of a formal schema and rigid type checking make these kind of
errors much easier to make, and with today's hardware they don't
become a problem until the database grows big.