[ZODB-Dev] ZEO client leaking memory?

[Toby Dickenson]

> > Ahhhh! Yes, BTrees do have a disk space overhead due to their
> > optimisation for incremental, not bulk indexing.
> 
> By disk-space overhead, I guess you mean lots of object 
> changes when you index
> something. I wonder if that could also be chewing through 
> RAM?

not as much as you are seeing.

> > * a size of 400 objects.
> > 
> > That means incremental garbage collection only seriously 
> kicks in when
> > there are at least 400 objects in the cache. In a well tuned system
> > under moderate memory pressure (in my experience) I would 
> expect each
> > cache to reach an equilibrium size of up to 2 or 3 times larger than
> > this.
> 
> What do you mean by object here? A whole BTree, a BTree bucket, etc?

Persistent objects derived from cPersistence.Persistent. For BTrees, a
bucket.

> I would guess a call to _p_jar.cacheMinimize() would override 
> this and just dump
> all objects out of RAM?

It certainly should do.

> > If you do switch to using it then I think these values will 
> be roughly
> > right for you. If you see cache sizes growing above 1200 then reduce
> > the time value. (its not very sensitive; I would try 10s next)
> 
> Well, how can I monitor and change these when I don't have a 
> web front end?

Dump the stats inside every time round your loop? This will definitely tell
you whether you have a real leak, or just poor ZODB cache management.

> > incrGC is designed to *maintain* a sensible pool of recently used
> > objects in memory, and not let that pool grow too large. It can only
> > do its job if it is called frequently.
> 
> Where does it get called in a normal Zope setup?

Just before every request.

(Going back to the top of this thread: I have found this isnt enough if the
request will be touching alot of objects; I also call it manually before
touching each bunch of objects)
 
> > Have a look at the source for the relevant bits of the zope 
> management
> > pages. Finding out exactly where the memory is going will be a big
> > clue.
> 
> Indeed... have to have a look.
> 
> man, and there was me hoping to do a Squishdot release today :-S
> 
> Chris
>