[ZODB-Dev] ZEO client leaking memory?

[Chris McDonough]

Note:  I know that solving the problem is job #1, but once you figure 
out what the problem is and come up with a reasonable way to solve it, 
it'd be very helpful to capture the information (ZODB-cache-tuning and 
memory-leak-related) that is flying by on this thread.  It's definitely 
underdocumented.

Toby Dickenson wrote:

>>>Ahhhh! Yes, BTrees do have a disk space overhead due to their
>>>optimisation for incremental, not bulk indexing.
>>>
>>By disk-space overhead, I guess you mean lots of object 
>>changes when you index
>>something. I wonder if that could also be chewing through 
>>RAM?
>>
> 
> not as much as you are seeing.
> 
> 
>>>* a size of 400 objects.
>>>
>>>That means incremental garbage collection only seriously 
>>>
>>kicks in when
>>
>>>there are at least 400 objects in the cache. In a well tuned system
>>>under moderate memory pressure (in my experience) I would 
>>>
>>expect each
>>
>>>cache to reach an equilibrium size of up to 2 or 3 times larger than
>>>this.
>>>
>>What do you mean by object here? A whole BTree, a BTree bucket, etc?
>>
> 
> Persistent objects derived from cPersistence.Persistent. For BTrees, a
> bucket.
> 
> 
>>I would guess a call to _p_jar.cacheMinimize() would override 
>>this and just dump
>>all objects out of RAM?
>>
> 
> It certainly should do.
> 
> 
>>>If you do switch to using it then I think these values will 
>>>
>>be roughly
>>
>>>right for you. If you see cache sizes growing above 1200 then reduce
>>>the time value. (its not very sensitive; I would try 10s next)
>>>
>>Well, how can I monitor and change these when I don't have a 
>>web front end?
>>
> 
> Dump the stats inside every time round your loop? This will definitely tell
> you whether you have a real leak, or just poor ZODB cache management.
> 
> 
>>>incrGC is designed to *maintain* a sensible pool of recently used
>>>objects in memory, and not let that pool grow too large. It can only
>>>do its job if it is called frequently.
>>>
>>Where does it get called in a normal Zope setup?
>>
> 
> Just before every request.
> 
> (Going back to the top of this thread: I have found this isnt enough if the
> request will be touching alot of objects; I also call it manually before
> touching each bunch of objects)
>  
> 
>>>Have a look at the source for the relevant bits of the zope 
>>>
>>management
>>
>>>pages. Finding out exactly where the memory is going will be a big
>>>clue.
>>>
>>Indeed... have to have a look.
>>
>>man, and there was me hoping to do a Squishdot release today :-S
>>
>>Chris
>>
>>
> 
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
> 
> ZODB-Dev mailing list  -  ZODB-Dev@zope.org
> http://lists.zope.org/mailman/listinfo/zodb-dev
> 


-- 
Chris McDonough                    Zope Corporation
http://www.zope.org             http://www.zope.com
"Killing hundreds of birds with thousands of stones"