[Zope-Coders] ThreadedAsync.LoopCallback

[Shane Hathaway]

Toby Dickenson wrote:
> Ive just stubled into a new dark corner of zodb; the 
> ThreadedAsync.LoopCallback module. It currently rebinds asyncore.loop to a 
> new function that also provides hooks for callback functions to be triggered 
> before entering a select loop. I am sure I dont fully understand how this is 
> used.
> 
> Previously Zope called asyncore.loop from z2.py to run the main select loop. 
> This was using the ThreadedAsync.LoopCallback version if ZEO is imported.
> 
> In Zope 2.7 this has changed - Zope now has its own main select loop in the 
> Lifetime.py module. It is like asyncore.loop with some additional exit 
> conditions to handle graceful shutdown.
> 
> Does Zope need to be calling the ThreadedAsync.LoopCallback hooks in its loop 
> too?

FWIW, here's my take on ThreadedAsync.  I think we can do away with it, 
getting simpler code in the process, with the only cost being an 
additional thread that runs anytime ZEO client code is used.  Here's an 
informal proposal I sent three weeks ago to an internal list:

"""
I just thought of something that might simplify the current ZEO 
implementation significantly.  Client-side ZEO uses asyncore in an 
unusual way: in order to mix with applications like Zope, it sometimes 
assumes there is no main loop running in another thread, so it runs a 
private loop inline.  This strategy is hard to explain and maintain, and 
it's spread over many parts of the code (as I recall).

What if, instead of relying on the application's main loop, ZEO clients 
ran a private asyncore loop in a ZEO-specific thread?  The client ZEO 
code would need only one such thread per process.  All communication 
with ZEO servers would be channeled through this thread (whereas today 
all server communication goes through the application's main loop). Here 
are some differences.

ThreadedAsync: no longer necessary.  Since the communication loop is 
always running, ZEO clients won't have to switch between asynchronous 
and synchronous mode.

Triggers: ZEO will only need one global trigger.  It will be set up when 
the communication loop is set up.

Interaction with asyncore: We'll have to use an alternate socket map. 
This is a rarely-used feature of asyncore, but it should be safe. 
Alternate socket maps allow different subsystems in an application to 
use asyncore without bumping into each other.  The downside is that all 
calls to set_socket(), add_channel(), and del_channel() need to specify 
the alternate socket map.

In retrospect, I think alternate socket maps running in secondary 
threads may be what Sam Rushing had in mind for problems like this. We 
need asynchronous communication, but it's hard to reuse the 
application's main loop.  So maybe we should use a private communication 
loop.  I've said that Twisted's core might improve the maintainability 
of ZEO, but this solution might achieve the same kind of improvement.
"""

I don't think it would be hard to implement at all.  We'd delete more 
code than we add.  No one is assigned to do it, though.

Shane