[ZODB-Dev] Optimize BTree node sizes?

[Laurence Rowe]

Hanno Schlichting wrote:
> Jim Fulton wrote:
>> On Nov 4, 2008, at 12:12 PM, Benji York wrote:
>>
>>> On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <jim at zope.com> wrote:
>>>> A few months back, there was a lot of discussion here about BTree
>>>> performance.  I got a sense that maximum BTree-node and bucket sizes
>>>> should be increased.  Does anyone have recommendations for new sizes?
>>> It'd be cool if the bucket size could be dynamic (say governed by an
>>> attribute on the BTree), but I suspect that is dramatically out of  
>>> scope
>>> for what you were planning on doing.
>> I have a list of projects I might try to do for 3.9.  One would be to  
>> make BTrees subclassable
>> and to modify BTrees to get these limits from class attributes.  But  
>> whether I get to this or not,
>> if there are more sensible defaults, it would make sense to use them.
> 
> Personally I think the numbers for the integer BTree's could be
> increased indeed, but for the object BTree's, a way to adjust the bucket
> sizes would be more important.
> 
> The problem for BTree's storing objects, is of course that you have no
> idea how large those objects are going to be. Sometimes they might be
> simple types with just a couple of bytes each, sometimes catalog brains
> with one kilobyte, but sometimes they might be actual content-like
> object of various megabyte in size.
> 
> In general there is no way of knowing what you are going to use the
> BTree's for. Making it possible for the application code to adjust the
> values for the particular kind of data, is probably better than guessing
> a better default value.

+1

The classic case here is comparing a catalogue metadata BTree with one 
containing persistent objects, where only the oid needs storing in the 
pickle.

Laurence