[ZODB-Dev] IIBTree.multiunion and list comprehensions

[Christian Robottom Reis]

On Tue, Dec 09, 2003 at 03:10:02PM -0500, Tim Peters wrote:
> >     multiunion(tree.keys())
> >
> > takes about the same time to run as
> >
> >     [value for set in tree.keys() for value in set] (*)
> >
> > My hypothesis is that even though multiunion() may run a bit faster,
> 
> It can run very much faster if the objects are in memory.  That's why it was
> added, BTW, for combining intermediate in-memory partial results into a
> final in-memory result.

What size of BTree would be necessary for the difference to actually be
noticeable? 

My application has BTrees with tens of thousands of objects, but nothing
above a hundred thousand objects. It's been quite fast with the
exception of when we union large numbers of sets, which is slow on the
first run (slowness which seems unavoidable given that the operation
potentially touches a large number of buckets in the tree).

I'm curious because I don't think I've ever heard of *very* large BTrees
being discussed on the list before. I assume you (and other Zopers) have
worked with some pretty gigantic ones; if you could throw up some
numbers, I'm guessing it would do wonders for my blood pressure <wink>.

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331