[ZODB-Dev] working with large databases

[Toby Dickenson]

On Thu, 28 Feb 2002 08:12:56 +0000, Chris Withers <chrisw@nipltd.com>
wrote:

>Greg Landrum wrote:
>>=20
>> The problem I'm encountering is that all of the spellings I can think =
of
>> result in the entire BTree ending up in memory.  This is not at all =
what
>> I'd like to have happen.
>
>I'd guess this is because all the BTree bits are ending up in the object=
 cache in RAM.
>
>I think Toby Dickenson has an alternate implementation of this Cache =
which might solve the
>problem.
>
>Bug him for it and see if it makes a difference...

ZODB loads objects into memory automatically on demand, but only moves
objects out of memory when you say it can. What you really need is to
change your loop so that it calls self._p_jar.cacheGC() every so
often.

How often should you call cacheGC? Thats where my patch comes in.
Without my patch you need to call it lots, and will pay a large
performance penalty.=20

With my patch it doesnt matter too much how often it is called, as
long as it is often enough. Calling every time round the loop will be
fine, maybe every 10 times or 100 times will save a few % of
performance.

Please let me know how this works out.

>...if it solves the problem, but him to merge it into the ZODB HEAD in =
CVS :-)

Im not sure I would like to be butted.

Toby Dickenson
tdickenson@geminidataloggers.com