[Zope3-dev] performance and BTrees

[Steve Alexander]

Marcus J. Ertl wrote:
> Hi!
> 
> I'm just playing around with the first version of my selfmade threaded
> forum! I have added 20.000 messages in 2.000 threads.
> 
> If I want to show the first 20 threads of my board, I call following
> methode:
> 
>     def messageList(self, start=0, end=20):
>         """Returns a list of messages"""
>         # optimzed by Christian Heimes
>         items = [self._messageInfo(itm)
>                     for itm in getAdapter(self.context,
> IZopeContainer).items()
>                         if ILNMessage.isImplementedBy(itm[1])
>                 ]
>         items.sort(self._cmpDatesDown)
>         return items[start:end]
> 
> But this takes to long! :-(

You should either cache the results of calculating messageList, or 
(better), maintain your own index of the date that messages are posted.


> It's even worse, if I try to view this
> folder with it's 2.000 Threads, because it goes rekursivly in all
> threads to count messages.
> (Messages are folderish objects, holding the replies, and so on.)

This has little to do with BTrees, and a lot to do with using an O(n^2) 
algorithm.


> So I have to problems:
> 
> 1.) The normal admin-view
> (http://gimli:8080/test/board/@@contents.html)  of a folder isn't good
> for realy large folders. It realy takes for ever to build up!

Of course. You should provide a different UI if you intend to put 2000 
objects in a folder, and want to view the contents page.

One general solution to this is that Folder objects could mark 
themselves with an IFolderThatContainsManyThings interface when the 
number of contained objects exceeds a particular threshold.

Then, contents views designed for such a large number of contained 
objects can be used instead of the usual folder views.


> 2.) I had to optimize the messageList-methode!
> 
> I would like to cut down the list before going over it, before sorting
> it. But I need to sort it by date, and filter out all not
> ILNMessage-Objects, before I can take my slice out! Even worse: In near
> future, there will be a filterstep more, because I had to add a
> workflow!
> 
> Is there a way to tell OOBTree, I'm only interested in a slice of it's
> .items()?

You should keep a separate BTree that indexes your content by date.

If someone wrote a DateIndex, you could use that, and define a view that 
uses the DateIndex to choose which contents to show.

--
Steve Alexander