[ZODB-Dev] ZEO+MultipleClients+ConflictErrors

[Benji York]

Alan Runyan wrote:
> Could you provide a bit more guidance?   Maybe an example.

Here's one: in certain circumstances people want to aggregate lots of 
incoming data, but doing the obvious thing of updating a data structure 
as new data comes in turns out to be too slow.  One way to handle that 
situation is to buffer incoming data and do a single aggregate write 
periodically (the approach taken by QueueCatalog).

Another simple one is wanting to keep up with a count, the naive way 
will get you in trouble (incrementing an attribute on an object), but 
using a the BTree.Length object helps a lot.

> Most people come at ZODB with previous experience in RDBMS.
> 
> How do they map SQL INSERT/UPDATE activities to ZODB data structures?
> In a way that does not create hotspot.
> 
> A few years ago people would say "pick a BTree implementation that maps
> to your data structures and use that."  But that is naive and pushes the
> hotspot into the BTree where concurrent updates are happening.

Naivety is relative.  Most people don't push their RDBMS or ZODB to 
their limits, so a straight-forward approach is often best.

Inserting object references into an IOBTree, for example, seems 
reasonable, being careful to choose good integer keys so there's little 
chance of keys conflicting.

If your insert rate is high enough that bucket splits cause too many 
conflicts, then you'll have to deal with that.  As I understand it one 
option is to choose your keys such that it is unlikely that two 
insertions happen in the same bucket.
-- 
Benji York
Senior Software Engineer
Zope Corporation