[ZODB-Dev] IIBTree.multiunion and list comprehensions

[Christian Robottom Reis]

On Wed, Dec 10, 2003 at 10:12:55AM -0500, Casey Duncan wrote:
> > I realize that first-time costs aren't really important for long-running
> > web applications; however, we're running a desktop system, and having
> > the initial operation execute slowly doesn't give a nice `first
> > impression', if you believe in such things.
> 
> Is there any way to defer this to later? Many applications (especially
> network bound ones like mail apps) use multiple threads so that the
> application can start fast and you can interact with it while it does
> the long process of loading lots of other stuff.

Well, we've put together the interfaces making sure that the application
startup will be fast. The startup speed is acceptable (it takes ~ 3
seconds for the main window to show up on the larges apps -- yeah, I put
a splash screen up). The problem is running those first queries once the
user is actually involved in a specific task (looking up all payments
for a certain supplier, for instance). Minimizing the objects retrived
by the database is something we do by carefully crafting the UI, and
providing indexes and optimized queries when we see something that
should be fast taking a long time.

The bottom line, which is obvious to me only now, is that the less stuff
you load from the ZODB, the better off you are, and the nice part is
that your user normally doesn't *want* to see a lot of stuff, *just the
stuff he wants*. 

OTOH, when you need to summarize information you have to be prepared to
put up progressbars and hope the user accepts the wait :-)

> The other option is to simply load less stuff. Is there any way to do
> this whole multiunion operation ahead of time and persist that
> (smaller) set? How many users share this database?

Around 20 users, though one user rarely uses the same application as 
another. 

Yes, one option is to store a special Set for this specific kind of
query (basically, it's for "foo != None" queries). 

The best part about using IC, ZODB and Python in general has been that I
have felt safe as I go forward -- if something's slow, I've found I only
need to tweak a certain index or use a special heuristic for a database
operation (inside IC mostly) and it's `all of a sudden fast again'.

For instance, yesterday I spent some time working on temporary indexes
that are useful when doing boolean queries ("foo == X and bar != Y", for
instance) -- if the first returns a small set and the second a very
large one, it's much cheaper to generate a temporary index and do the
bar query inside it. It would be fun if I could find a way to slice the
BTree and not even need a temporary index, but since I need to slice by
values, it remains to be seen what I'm going to do to implement *that*.

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331