[ZODB-Dev] Zope, ZEO, BTrees, and conflicts

[sean.upton@uniontrib.com]

Forgive my ignorance here.  I have a Zope application that writes a lot of
very small objects into a container subclassed from BTreeFolder (327,000, to
be more precise).  From my first impressions working with stress tests for
this application, I think that performance using the objects is good, though
performance bulk-rebuilding all the objects is slow, but acceptably so (and
mainly that is because I'm not running it on really fast hardware).  My head
is toying with the idea of distributing this across several ZEO clients, but
I'm betting there would be a lot of issues associated with doing that.

Perhaps I'm being a bit preemptively concerned, since I haven't had any
problems here yet (or even tried anything - I want to get others more
informed opinions here first), but I'm curious as to what possible
activities could cause either type of ZODB conflict with multiple threads or
clients writing to the same BTree, like in the case of 2 ZEO clients writing
to the same BTreeFolder, like:

#client node 1 does this:
self._tree['1'] = obj

#while simultaneously (or close to it) client
# node 2 does this, provided keys are kept unique
# across all nodes:
self._tree['2'] = obj

I'm really thinking that in theory, it would be nice to put distribute some
activity - i.e. BTreeFolder._setOb() - simultaneously across several ZEO
client nodes to speed up my application.  Even if I could do this - the
objects I add are CatalogAware, and I call index_object() - I worry that I
wouldn't be able to, since I assume I might run into similar issues with the
IOBTrees used in ZCatalog indexes.

I know there was some sort of similar discussion back in Feb. on zope-dev,
but I didn't quite understand all the details of what I found in the list
archives.  I'm just trying to get a firm grasp on the issues associated with
write-intensive applications using the Zope/ZODB (primarily in regard to a
large bulk add-and-reindex of objects), and potential solutions.

Sean

=========================
Sean Upton
Senior Programmer/Analyst
SignOnSanDiego.com
The San Diego Union-Tribune
619.718.5241
sean.upton@uniontrib.com
=========================