[Zope-dev] zope2/zodb cache size question

[Jürgen Herrmann]

On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton <jim at zope.com> wrote:
> On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann
> <Juergen.Herrmann at xlhost.de> wrote:
>>
>> hi there!
>>
>> today i ran a check script which iterates over approx 150000 objects
>> and does some sanity check calculations on these. during this loop
>> i saw the zope process use up to about 4.5gb memory. the database
>> has ~3.5million objects in it.
>>
>> i set the zodb cache size for the mount point in question to 10000
>> objects. obviously this limit is not honoured during one transaction:
> 
> Right.
> 
>> so two questions here:
>> - would the byte-limited zodb cache setting help here?
> 
> No. The limits aren't really limits. They are more like
> suggestions. :)  In particular, they are only enforced at
> transaction (or subtransaction) boundaries, when a connection is
> opened or closed, or when applications call certain APIs explicitly.
are these apis exposed somewhere? calling these periodically sounds
cleaner than the aproach below.
> 
>> - if no - how can i iterate over a big set of objects without
>>  forcing them to stay in the cache for the whole transaction?
>>  after all i just need each object once during the iteration.
> 
> As you're iterating through the objects, if you know you aren't going
> to need an object again or otherwise think it would be OK to free it
> from memory, you can call _p_deactivate on it. If the object hasn't
> been modified, then it's state will be released and it will become a
> ghost.
as my objects contain references to OOBTrees, do i also have to call
_p_deactivate() on these explicitly, or is it enough to call it on the
containing object?

thanks for your help!
best regards, jürgen
-- 
>> XLhost.de - eXperts in Linux hosting ® <<

XLhost.de GmbH
Jürgen Herrmann, Geschäftsführer
Boelckestrasse 21, 93051 Regensburg, Germany

Geschäftsführer: Volker Geith, Jürgen Herrmann
Registriert unter: HRB9918
Umsatzsteuer-Identifikationsnummer: DE245931218

Fon:  +49 (0)800 XLHOSTDE [0800 95467833]
Fax:  +49 (0)800 95467830

WEB:  http://www.XLhost.de
IRC:  #XLhost at irc.quakenet.org