[ZODB-Dev] Re: From memory Problems to Disk Problems :-(

[Chris Withers]

"Barry A. Warsaw" wrote:
> 
>     CW> The import script only uses 100Mb odd of RAM now, but
>     CW> something that the BDB tables involved are doing is causing
>     CW> the cache to jump up to 536Mb :-(
> 
> Which cache are you talking about?

The one on the far right of the 'swap' line in top...

> Have you read the tuning pages for BerkeleyDB?  

Erk.. there are tuning pages? I should probably read them...

> I'm not sure.  Sleepycat claims that they can handle big objects just
> fine.  I suppose I'd err on the side of lots of smaller commits.

That's what I thought, I'll give it a try. I've just put some timing metrics
into my script so I can play around with how long it takes to index documents
based on various permutations of object per commit.
May you can do the same with yours and we can compare? Is there any way to tell
the absolute size of a transaction as you commit? That would be interesting to
compare...

cheers,

Chris