[ZODB-Dev] bsddb3 mass creation problem.

[Christian Robottom Reis]

On Thu, 17 May 2001, Michel Pelletier wrote:

> I don't know the solution to your problem, but I do know that calling
> commit(1) every time you add an object is not more efficient than calling
> commit() every time.  You should call commit(1) after every N objects are
> added, N being probably on the order of at least a hundred (depending on
> your objects).  just change the second to last line to:

Michel, it seems that, apart from that, it's not a very good idea to keep
that many sub-transactions open and _then_ commiting them all at once.
Though the memory usage is controlled in that fashion (those subobjects
can be thrown away), all those sub-commits will be committed on the final
commit(), which for some reason is eating up locks like mad. Of course,
there are reasons why an application might want to keep that many
sub-transactions created, but I'm just testing here, so what do I care?
:-)

Doing a final commit() every 100 object creates keeps the lock usage down,
which is great. I've managed to create 20,000 objects in 2:30, which is
great performance for writing (though linear, non-concurrent). This is
excellent performance, AFAIC.

Most importantly, using bsddb3 has kept the database open times low, which
is great for my application's current configuration. However, I'll
probably move to a zeo-based configuration, which keeps the storage open
and ready and avoid the open times altogether.

Take care,
--
/\/\ Christian Reis, Senior Engineer, Async Open Source, Brazil
~\/~ http://async.com.br/~kiko/ | [+55 16] 274 4311