[ZODB-Dev] PersistentMapping good for large numbers of objects?

[Shane Hathaway]

Tim Peters wrote:
> [Shane Hathaway]
> 
>>How about a combination of the above suggestions: every operation that
>>mutates a BTree could look for an attribute of the BTree called
>>"length".
> 
> Wow -- I've been looking for ways to slow BTrees, and this one didn't even
> occur to me <wink>.

I don't think we're thinking of the same idea. :-)  Only Python-level 
operations would invoke the callback, and only if the callback exists. 
Along with being more reliable and less to remember, I think it could 
actually be faster than maintaining an independent length object like we 
do now, since we often fall into the update() trap discussed earlier.

>>If it's there, the mutating operation could call length.changed
>>(deltaLength).  Also, the __len__ slot could delegate to the length
>>object.  That way, we don't have to wrap the BTree just to get a
>>minor (but nice) feature.
> 
> A practical snag is that "a BTree" doesn't exist as such -- it's a
> (potentially very large) collection of BTree and Bucket nodes, and nothing
> at all distinguishes the root BTree node (which alone is "the BTree" from a
> program's POV, but that's an illusion, and mutations *actually* occur in
> Buckets, not in BTree nodes).

Yep.  So what I have in mind is passing around an "int *deltaLength" in 
BTree mutating functions.  The mutators would modify *deltaLength as 
they work, then at the end of the operation (involving many nodes), that 
deltaLength (which exists only on the stack) would be given to some 
other object.  It's similar to the way BTrees used to work: they stored 
a length in the root BTree node.

This assumes that for every BTree-mutating operation, there is a 
function that gets called externally but never internally.  The external 
function is where we'd call length.change().  It's common for Python 
extensions to separate external and internal functions, but I haven't 
looked at whether the BTrees code uses that pattern.  If it doesn't, 
this idea is no good. ;-)

Shane