[ZODB-Dev] bsddb3 mass creation problem.

[ender]

On Thursday 17 May 2001 15:54, Christian Robottom Reis wrote:
>>On Thu, 17 May 2001, Michel Pelletier wrote:
>>> I don't know the solution to your problem, but I do know that calling
>>> commit(1) every time you add an object is not more efficient than calling
>>> commit() every time.  You should call commit(1) after every N objects are
>>> added, N being probably on the order of at least a hundred (depending on
>>> your objects).  just change the second to last line to:
>>
>>Michel, it seems that, apart from that, it's not a very good idea to keep
>>that many sub-transactions open and _then_ commiting them all at once.
>>Though the memory usage is controlled in that fashion (those subobjects
>>can be thrown away), all those sub-commits will be committed on the final
>>commit(), which for some reason is eating up locks like mad. Of course,
>>there are reasons why an application might want to keep that many
>>sub-transactions created, but I'm just testing here, so what do I care?
>>
>>:-)
>>
>>Doing a final commit() every 100 object creates keeps the lock usage down,
>>which is great. I've managed to create 20,000 objects in 2:30, which is
>>great performance for writing (though linear, non-concurrent). This is
>>excellent performance, AFAIC.

i'm not familiar with the bsddb3 storage details, but with other storages the 
locks will happen irrespective of sub trans v. trans, there part of the 
storage layer for concurrent usage (threads) . hence doing a subtrans every 
100 objects should also keep lock usage down. doing sub trans should lower 
your memory requirements and allow for overall integrity during the add 
process

kapil