[ZODB-Dev] design issue: job queue is concurrency bottleneck

[John Belmonte]

Tim Peters wrote:
> John, how many things do you expect to store in one of your job queues?  If
> it's only a few hundred (or less), you can use a Bucket instead of a BTree.
> A single Bucket won't suffer a write conflict unless multiple transactions
> try to delete, or add, or change the value associated with, the same key.

I'm expecting a few orders of magnitude more items in the job queue than 
that.

What I'm leaning towards at this point, given that I have a fixed set of 
constant times in the future to schedule a job (now, now + 1 hour, now + 
3 days, etc.), is to make a separate queue for each time constant.  Then 
I'll only be adding things to the end of each queue, eliminating the 
need for random insertions.  I'll have to make some custom list 
container with conflict resolution for appending items.  Only one 
process is allowed to remove items so I can ignore that part.  I'll 
probably want to chunk the list so that its not all in memory, as 
normally only items near the head will be accessed.

-John


-- 
http:// if  ile.org/