[ZODB-Dev] IIBTree.multiunion and list comprehensions

[Christian Robottom Reis]

On Wed, Dec 10, 2003 at 09:27:51AM -0500, Casey Duncan wrote:
> > What I'm doing is collating all the values in all the sets in my BTree.
> > This BTree holds as values IITreeSets containing a number of integer
> > OIDs. When I want to pick up *all* the values in the BTree, I need to
> > join all these IITreeSets together and produce one big set. The fact
> > that it's unique doesn't really matter in my case given that any given
> > OID appears only once [as a value].
> 
> In that case it may be cheapest for you not to do the union up front
> at all. How about creating a lazy iterator that just walks down every
> member of every set in the tree? The big downside of that is that you
> can't really do anything with the result set besides iterating it,
> such as sorting or intersecting it.

This is probably a good idea (though it requires Python 2.2, which we
haven't yet in IndexedCatalog). You last phrase confused me, however: do
you mean I *can* sort and intersect the result set? If so, it would be
neat, since what I want to do in many cases is intersect this with
another set (resulting from a boolean query).

My main worry with *that*, however, is that intersecting will end up
having a similar cost to doing multiunion, since the cost I'm concerned
about is the first-run cost (IOW, the cost of unpersisting the actual
sets) and AFAIHS multiunion is too fast to notice once the sets are
in-memory.

I realize that first-time costs aren't really important for long-running
web applications; however, we're running a desktop system, and having
the initial operation execute slowly doesn't give a nice `first
impression', if you believe in such things.

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331