[ZODB-Dev] [ZEO] rather long periods of unresponsiveness

[Jeremy Hylton]

On Sat, 18 Dec 2004 18:27:54 +0100, Dieter Maurer <dieter at handshake.de> wrote:
>   Only in the "vote" (end of first commit phase),
>   ZEO calls the storage's "tpc_begin" (and thereby acquires
>   the storage's commit lock) and then transfers the changed
>   data from the "CommitLog" to the storage.
>   Depending on the size of the transaction and the number
>   of affected objects, this can take a long time and
>   ZEO is irresponsive during this time.
> 
> What do you think about executing "vote" (more precisely
> "_vote") in a separate thread when the transaction is sufficiently
> large (with respect to size or number of modified objects)?
> This would allow the main ZEO thread to
> process other unrelated requests in parallel.

In principle, it sounds reasonable.  There are already several methods
that are run in separate threads so that the asyncore mainloop
continues to run.  The primary risk is that you have multiple threads
involved in the management of the commit lock.  I don't think it's a
big problem; the second thread will always start with the lock
acquired and release it when it is finished.

Note that the long delay at vote may also be a problem because it is a
synchronous call.  The client will block waiting for vote to return,
regardless of whether the server runs it in another thread.  The
blocking could be unfortunate if another thread in the same app server
wants to load an object; the load will be blocked until the vote
finishes, even if the server could be responsive.

We looked at changing vote to be asynchronous about a year ago.  There
may even be a branch with a partial implementation of the feature,
although I've got no recollection of the branch's name.  It might be
good to consider both changes, although the asynchronous vote is a
protocol change and probably more delicate than the threading change.

Jeremy