[ZODB-Dev] ZEO client leaking memory?

[Jim Fulton]

Toby Dickenson wrote:
> 
> On Thu, 4 Oct 2001 13:06:03 -0400 (EDT), Jeremy Hylton
> <jeremy@zope.com> wrote:
> 
> >[More on this later.]
> 
> Last time I looked at this my final thoughts on "the real problem" was
> that the size limits imposed by the ZODB are too soft.
> 
> the amount of work that cPickle cache performs in trying to remove old
> objects from the cache is proportional to the amount by which the
> actual cache size exceeds the specified limit. This means that in an
> application that touches alot of objects, the cache will eventually
> reach an equilibrium with a size which is roughly proportional to the
> average number of objects touched between calls to incrGC.
> 
> The code that controls this equilibrium is commented /* Decide how
> many objects to look at */ in cPickleCache.c. At the worst case, the
> cache has to be 10 times larger than the specified limit before
> cPickleCache checks every one one them. I have managed to force ZODB
> into this state; under high memory pressure the process does indeed
> grow to roughly 10 times its size under low memory pressure.

Excellent analysis.

> My chosen solution at the time (which has since proven itself in
> action) was to sprinkle more calls to incrGC within the
> object-intensive code.

What do you mean by object-intensive code?  Was this application
code? This probably increases the chance of "read conflicts", but
that might not be a problem for you. (I really wish we had multi-version
concurrency control.)

> This keeps the average number of objects
> touched between calls to incrGC constant, and therefore memory usage
> is constant. Fortunately individual calls to incrGC are fairly fast
> when called frequently.

I'd love to get suggestions on improving the cache management
algorithm.

Jim

--
Jim Fulton           mailto:jim@zope.com       Python Powered!        
CTO                  (888) 344-4332            http://www.python.org  
Zope Corporation     http://www.zope.com       http://www.zope.org