[ZODB-Dev] Runaway cache size

[Gary Poster]

I'm the primary zc.catalog author.

He is running read-only, so savepoints shouldn't be very pertinent.

The original poster is correct that the cache is cleaned out at  
transaction boundaries (or savepoints, as Alan said).

I don't have experience with running the db in readonly mode in  
production.  That said, if you have obvious boundaries for your  
searches, I suspect that beginning a transaction at the boundary would  
probably do the trick, and also work (with invalidations and the like)  
if you ever have a readonly connection to a ZEO database server that  
is getting updates from another source.

While it's interesting to contemplate the knob that Alan mentioned,  
I'm reasonably confident that this logic belongs in the app, not the  
index or catalog.

Gary

On Jul 31, 2008, at 11:55 AM, Alan Runyan wrote:

> I believe it is common to have a theshold (say every 10000) objects to
> do a savepoint
> and then you can run cacheMinimize after the savepoint.
>
> surprised the zc.catalog does not provide this sort of knob. maybe it
> was a problem
> in the original ZCatalog implementation?
>
> cheers
> alan
>
> On Wed, Jul 30, 2008 at 1:21 PM,  <AFoglia at princeton.com> wrote:
>>
>>        I have a database consisting mainly of an IOBTree of  
>> ~700,000 items
>> (each persistent mappings), and an zc.catalog indexing them by one  
>> of the
>> mapping's fields (a date).  I am opening the FileStorage in  
>> read_only mode.
>> For each day in the index, I get the day's mappings and read the  
>> contents
>> of another field.  As I go through each day, the memory usage  
>> explodes (over
>> 32 GB).  Is there a way to configure the cache to automatically  
>> keep itself
>> under the value of the cache size parameter?
>>
>>        To avoid this problem in most cases, I wrap the IOBTree in  
>> another
>> object which does nothing more than call db.cacheMinimize after  
>> every 10000
>> items are iterated over.  But for random access, that's not an  
>> option.
>>
>>        From reading the archives, it sounds like cache cleaning  
>> does not
>> happen while running a transaction.  Is that my problem?  I'm in  
>> read-only
>> mode, so I can't perform a transaction anyway, but could the ZODB  
>> think I
>> want to?  Or is my problem something else?
>>
>>        Maybe if someone can point me to a description of how the  
>> caching
>> works.
>>
>> --
>> Anthony Foglia
>> Princeton Consultants
>> (609) 987-8787 x233
>> _______________________________________________
>> For more information about ZODB, see the ZODB Wiki:
>> http://www.zope.org/Wikis/ZODB/
>>
>> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
>> http://mail.zope.org/mailman/listinfo/zodb-dev
>>
>>
>
>
>
> -- 
> Alan Runyan
> Enfold Systems, Inc.
> http://www.enfoldsystems.com/
> phone: +1.713.942.2377x111
> fax: +1.832.201.8856
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev