[ZODB-Dev] Optimize BTree node sizes?

[Hanno Schlichting]

Jim Fulton wrote:
> On Nov 4, 2008, at 12:12 PM, Benji York wrote:
> 
>> On Tue, Nov 4, 2008 at 12:01 PM, Jim Fulton <jim at zope.com> wrote:
>>> A few months back, there was a lot of discussion here about BTree
>>> performance.  I got a sense that maximum BTree-node and bucket sizes
>>> should be increased.  Does anyone have recommendations for new sizes?
>> It'd be cool if the bucket size could be dynamic (say governed by an
>> attribute on the BTree), but I suspect that is dramatically out of  
>> scope
>> for what you were planning on doing.
> 
> I have a list of projects I might try to do for 3.9.  One would be to  
> make BTrees subclassable
> and to modify BTrees to get these limits from class attributes.  But  
> whether I get to this or not,
> if there are more sensible defaults, it would make sense to use them.

Personally I think the numbers for the integer BTree's could be
increased indeed, but for the object BTree's, a way to adjust the bucket
sizes would be more important.

The problem for BTree's storing objects, is of course that you have no
idea how large those objects are going to be. Sometimes they might be
simple types with just a couple of bytes each, sometimes catalog brains
with one kilobyte, but sometimes they might be actual content-like
object of various megabyte in size.

In general there is no way of knowing what you are going to use the
BTree's for. Making it possible for the application code to adjust the
values for the particular kind of data, is probably better than guessing
a better default value.

Hanno