[ZODB-Dev] working with large databases

[Toby Dickenson]

On Thu, 28 Feb 2002 10:39:13 -0800, Greg Landrum
<greglandrum@earthlink.net> wrote:

(CC Lawrence)

>>How often should you call cacheGC? Thats where my patch comes in.
>>Without my patch you need to call it lots, and will pay a large
>>performance penalty.
>>
>>With my patch it doesnt matter too much how often it is called, as
>>long as it is often enough. Calling every time round the loop will be
>>fine, maybe every 10 times or 100 times will save a few % of
>>performance.
>
>I had to apply parts of the patch by hand because the versions of DB.py =
and=20
>Connection.py in the diff file do not correspond to any versions I could=
=20
>actually find.

Thanks for persevering with this. The diffs are produced from our
private cvs repository. Im suprised its not close enough to the
standard 2.4.1 for the diffs to apply cleanly, although the revision
numbers will be different.

>  The result builds and works under linux.  On windows it=20
>dumps core as soon as I try and do a commit.

Interesting. We do some development under Windows here so I know it
works under low load, but it hasnt been pushed. Lawrence, do you have
some time to try this?

>  This makes me kind of sad=20
>because my big memory machine is running Windows.  I'd love to be able =
to=20
>trace this down, but getting a full debug build of everything I need is =
a=20
>nightmare I don't want to relive right now.

I would be happy to look into this if you have a simple test case.

>I'm happy to report that, on linux, using Toby's patch and a call to=20
>_p_jar.cacheGC() every thousand times through the loop reduces memory=20
>requirements substantially.  The performance penalty is about 15% =
(that's a=20
>rough number), which isn't half bad.  Color me pleased.

15% relative to what? the version that never called cacheGC and sucked
memory? yes that isnt half bad, but its not as good as I would like.

>Now if I could just make it work on windows...

Indeed.

Toby Dickenson
tdickenson@geminidataloggers.com