[ZODB-Dev] ZEO client hangs when combined with other asyncore code

[Jim Fulton]

Tim Peters wrote:
> ...
> 
> [Tim Peters]
> 
>>>asyncore gives me a headache.
> 
> 
> [Paul Boots]
> 
>>Same here
> 
> 
> Then it's time to admit that ZEO's attempts to mix threads with asyncore
> give me migraine headaches <0.5 wink>.
> 
>>>I wonder whether this could be the problem:  Paul said he's calling
>>>ZEO "from within the proxy code", but it sounds like the proxy code
>>>itself runs "as a side effect" of asyncore callbacks.  If the flow is
>>>like this:
>>>
>>>  asyncore mainloop invokes POP3 proxy code
>>>      POP3 proxy code makes a synchronous ZEO call
>>>
>>>then I figure the app may well hang then:  the thread running the
>>>asyncore mainloop is still running a POP3 proxy callback, waiting for a
>>>response that can never happen until the asyncore mainloop gets control
>>>back (in order to send & receive ZEO messages).
> 
> 
>>I think that's exactly how the Proxy runs, we use asynchat and the
>>'line_terminator' to trigger a callback, so it appears the code runs
>>'magically' at first glance.
> 
> 
> I never used asynchat (& ZEO doesn't either), so can't guess whether it's
> contributing "new" complications.  ZEO's control flow is murky to me too.  I
> _think_ (but may well be wrong) that ZEO expects asyncore to be running in a
> different thread than the thread(s) application code using ZEO clients
> is(are) running in.  Maybe someone who understands this better than I will
> jump in with a revelation.

ZEO has 2 modes, synchronous and asynchronous.

In asynchronous mode, ZEO expects a "asyncore main loop" to be running
in it's own thread.

A basic constraint of asyncore (and Twisted) programming is that
I/O handlers must perform their tasks very quickly.  Generally,
they are expected to do short tasks, moving small amounts of data
around at a time.  Generally, this means that they should not
be calling application code directly.  This is why Zope executes
application logic in separate application threads and limits work
done by asyncore handlers to simple I/O.

...

> IMO/IME, asyncore is a poor fit for applications where the callbacks are
> "fancy", or even where they may just take a long time to complete (because
> the asyncore mainloop is unresponsive for the duration).  So if I had to use
> asyncore (I've never done so on my own initiative <wink>), I'd gravitate
> toward a work-queue model anyway, where threads unfettered by asyncore
> worries do all "the real work"-- especially on Windows, which loves to run
> threads --and where asyncore callbacks do as little as possible.

Agreed.

This is exactly the model that Zope uses.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org