[Zope3-dev] Re: performance and BTrees

[Christian Heimes]

Steve Alexander wrote:
> You'd want to use the event service.
> Write a DateIndex and subscribe it to receive IObjectAddedEvents.
> When the object that is added is an IYourMessageThread, index its id 
> (its path, I guess) against the date it was created.

I thought of using hub ids instead of the path and IHubEvents. Threads 
can be removed or moved around to other boards. Objects keep their hub 
id, don't they?

FYI: Marcus and I thought of this structure for the forum:
forum 1..n boards
boards 0..n threads
boards 0..n boards
threads 1..n messages

> If you have lots of other objects being added in your system, you might 
> want to send an IMyMessageThreadAddedEvent when your message threads are 
>  created. Then, you subscribe your DateIndex to just those events.

Having our own events is a nice idea. Thanks!

> BTrees maintain their items in the sort order of their keys.

Is it possible to have a custom order? Marcus needs to sort the values 
by date. My idea was to subclass btree and change some methods. A sort 
function must be provided as an argument OOBTree(myCmpFunction). All 
tree changing methods first call it's super and then the sort function 
on the tree itself.

> I'd have just one central date index for all your threads.
> If you also maintain an index of thread->message, you will be able to 
> very quickly get the messages for a particular thread sorted by date.
> 
> Otherwise, if you want to maintain a BTree with each message thread, you 
> can have one subscriber for all your threads that dispatches the event 
> to the particular message thread that needs to know about it.

For me I would prefer the first alternative.

Other question:
Marcus wants to have a non-random unique id for threads and messages 
which is as low as possible, too. This method is my first idea to make a 
sql like sequence. It stores also the hub id to look up the path and 
object in an easy way. I'm not shure wether this method is thread safe 
or not.

def genThreadId(self, obj):
     """
     tid2hubid and hubid2tid are both IIBTrees
     """
     hubid = query_the_hub_method(obj)
     try:
         tid = self.tid2hubid.maxKey()+1
     except ValueError:
         # emptry tree
         tid = 1
     # do I need this while loop to be thread safe?
     # is this whole function thread safe?
     while not self.tid2hubid.insert(tid, hubid):
         tid+=1
     self.hubid2tid.insert(hubid, tid)
     return tid

Alternative version with lock:

from thread import allocate_lock
genThreadId_writelock = allocate_lock()

def genThreadId(self, obj):
     """
     tid2hubid and hubid2tid are both IIBTrees
     """
     hubid = query_the_hub_method(obj)
     genThreadId_writelock.acquire()
     try:
         tid = self.tid2hubid.maxKey()+1
     except ValueError:
         # emptry tree
         tid = 1
     self.tid2hubid.insert(tid, hubid)
     self.hubid2tid.insert(hubid, tid)
     genThreadId_writelock.release()
     return tid

I suppose the version with thread.lock is the better one.

Christian