[ZODB-Dev] How to update an object in a multithreading application?

[Vincent Pelletier]

Le lundi 19 mars 2012 20:59:24, Sebastian Wain a écrit :
> Indeed I want to sync the same BTree between threads. I am adding
> persistence and acknowledgement to the Python queue.

I don't understand what you mean by:
- "sync": ZODB is probably not a good backend to provide synchronisation as a
  feature, ie it's not a "lock server" and does what it can to avoid locking
  (optimistic transaction, working best on use cases where conflicts are
  unlikely by design).
- "same BTree" vs. "Python queue": I don't get the link here. A ZODB BTree
  involved in an alternate implementation of Python's "Queue" module ?

> Is that expensive in terms of performance?

Hard to tell without knowing the implementation and (maybe) use cases.

Some points out the top of my head which should be worth considering:
- In one of my projects, Pure Python Queue.Queue showed (single producer,
  single consumer) bad lock contention on the producer side (!). I'm talking
  about 30% of execution time being spent waiting on lock on producer side,
  and 40% on consumer side, IIRC, with quite fast code executed on each
  side. This is because of Queue API providing current size in number of
  items, which requires serialises all accesses.
  I the case of the queue element counter protected by that producer-hurting
  lock, using BTree.Length makes sense (warning: I didn't think it through,
  and it might very well depend on your implementation).
- Note that ZODB's typical backend (ZODB.FileStorage) keeps a history of all
  object states it ever saw. Depending on your usage, you might not need that
  history to be preserved, and using an "history free" backend would serve you
  better. OTOH, doing that prevents conflict resolution from working (removing
  all interest from using BTree.Length). Packing can help, but on a large
  enough database (compared to available RAM) it might take a significant
  amount of time.
- As I said above, design to avoid hot spots: if two concurrent transactions
  are likely to alter the same object you will have poor performance, and
  difficult decisions to take to get out of it (if constrained by backward
  compatibility). When you must have a hot spot, design so that solving its
  conflicts:
  - is feasible (ie, there is one answer to enough conflict cases to be
    actually useful)
  - can be done without accessing any other object

Regards,
-- 
Vincent Pelletier

PS: I believe HTML mails are considered bad, especially on mailing lists. 
Please use raw text.