[ZODB-Dev] IIBTree.multiunion and list comprehensions

[Tim Peters]

[Christian Robottom Reis]
> [I've been messing around with optimizations to query results in
> IndexedCatalog, and I've run into a result that's puzzled me a bit,
> but since I'm now over it, I'm sharing here.]
>
> ISTM that, when accessing a persisted BTree that holds Sets as
> its values, using multiunion (I've tested in particular with IIBTree's
> version) has no significant speed advantage over using a list
> comprehension to glue the sets together. IOW, assuming tree is a
> yet-to-be-unpersisted BTree with a large number of items stored,
>
>     multiunion(tree.keys())
>
> takes about the same time to run as
>
>     [value for set in tree.keys() for value in set] (*)
>
> My hypothesis is that even though multiunion() may run a bit faster,

It can run very much faster if the objects are in memory.  That's why it was
added, BTW, for combining intermediate in-memory partial results into a
final in-memory result.

> we're bound by the retrieval of buckets from the persistence
> mechanism, which is [comparatively] slow.

Sounds right to me -- and disk I/O is much more expensive than in-memory
operation.