[ZODB-Dev] Re: ZODB Benchmarks

[Russ Ferriday]

Quick note...

Smaller buckets, fewer conflicts, more overhead on reading and writing.
Larger buckets, more conflicts, less overhead on reading and writing.
One bucket ... constant conflicts.

I'd bet that the additional tree with tiny buckets would be best.  
Transfer them into the normal tree once the overhead starts rising.

How to transfer them?  You would not want a single transaction to  
take the hit for the whole transfer, so have a low and high water  
mark. When hitting HWM, transfer only until LWM is reached. Or, just  
focus on transferring some items out of a single tree, to avoid the  
cost of tree rebalancing on the additional tree at the same time as  
rebalancing on the main tree.

Sounds like a fin project,

--r.

On 1 Nov 2007, at 21:00, David Binger wrote:

>
> On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote:
>
>> David Binger <dbinger <at> mems-exchange.org> writes:
>>
>>>
>>>
>>> On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote:
>>>
>>>> Ie we perhaps look at a catalog data structure
>>>> in which writes are initially done to some kind of queue then moved
>>>> to the
>>>> BTrees at a later point.
>>>
>>> A suggestion: use a pair of BTrees, one with a high branching factor
>>> (bucket size)
>>> and one with a very low branching factor.  Force all writes into the
>>> tree with little
>>> buckets.  Make every search look in both trees.  Consolidate
>>> occasionally.
>>
>> An interesting idea.  Surely we need the opposite though, and that  
>> is an
>> additional BTree with a very large bucket size, as we want to  
>> minimize the
>> chance of a bucket split when inserting?  Then we occasionally  
>> consolidate and
>> move the items in the original BTree with the regular bucket size/ 
>> branch factor.
>
> You may be right about that.  Conflict resolution makes it harder for
> me to predict which way is better.  If you don't have conflict  
> resolution
> for insertions, then I think the smaller buckets are definitely better
> for avoiding conflicts.  In either case, smaller buckets reduce the
> size and serialization time of the insertion transactions, and that  
> alone
> *might* be a reason to favor them.  I think I'd still bet on smaller
> buckets, but tests would expose the trade-offs.
>
>
>
>
>
>
>
>
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev

Russ Ferriday
russ at indepthoptics.com
office: +44 118 3217026
mobile: +44 7789 338868
skype: ferriday



-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zodb-dev/attachments/20071101/89cc6ee1/attachment.htm