[ZODB-Dev] ZEO client leaking memory?

[Toby Dickenson]

On Thu, 4 Oct 2001 13:06:03 -0400 (EDT), Jeremy Hylton
<jeremy@zope.com> wrote:

>[More on this later.]

Last time I looked at this my final thoughts on "the real problem" was
that the size limits imposed by the ZODB are too soft.

the amount of work that cPickle cache performs in trying to remove old
objects from the cache is proportional to the amount by which the
actual cache size exceeds the specified limit. This means that in an
application that touches alot of objects, the cache will eventually
reach an equilibrium with a size which is roughly proportional to the
average number of objects touched between calls to incrGC.

The code that controls this equilibrium is commented /* Decide how
many objects to look at */ in cPickleCache.c. At the worst case, the
cache has to be 10 times larger than the specified limit before
cPickleCache checks every one one them. I have managed to force ZODB
into this state; under high memory pressure the process does indeed
grow to roughly 10 times its size under low memory pressure.


My chosen solution at the time (which has since proven itself in
action) was to sprinkle more calls to incrGC within the
object-intensive code. This keeps the average number of objects
touched between calls to incrGC constant, and therefore memory usage
is constant. Fortunately individual calls to incrGC are fairly fast
when called frequently.



Toby Dickenson
tdickenson@geminidataloggers.com