[ZODB-Dev] design issue: job queue is concurrency bottleneck

[Christian Robottom Reis]

On Thu, Mar 04, 2004 at 12:04:36AM -0500, John Belmonte wrote:
> I finally got through the first round of concurrency testing on my app. 
>    Despite being a high-write application, virtually no write conflicts 
> occur-- with one major exception.  I have a job queue, which is a BTree
> keyed by scheduled job time.  It's the only BTree in my app that doesn't 
> have random keys.  Incoming requests to my server tend to generate a few 
> jobs, and unfortunately they must be scheduled at various constant time 
> periods in the future.  As a result, simultaneous requests generate jobs 
> scheduled for approximately the same time, mapping to the same area of 
> the BTree, and causing write conflicts.
> 
> Short of adding some random offset to scheduled job times, which I feel 
> would not be very effective (depends on BTree size, etc.), does anyone 
> have ideas for designing around this bottleneck?

So your data structure doesn't take into account the fact that multiple
jobs can be allocated to the same timeslot?

Have you considered trying to add the key, catching ConflictError, and
trying again adding a small time offset?

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 261 2331