[ZODB-Dev] Storage Interfaces

[David Pratt]

Jim Fulton wrote:
> 
> On Apr 27, 2007, at 9:30 PM, David Pratt wrote:
> 
>> Hi Jim. I have been quite interested in this also so have experimented 
>> and continue on this also. The way I am doing the networking it 
>> similar to ngi I have put this together before you defined these 
>> interfaces.
> 
> I'm not sure which "this" you're referring to. I assume you mean using 
> Twisted for ZEO.  I don't see that that has much to do with the new 
> storage interfaces.

Yes, Twisted for ZEO. No, I was referring to the interfaces you were 
setting up - the interfaces to ngi.

> 
> In what way is your networking similar to ngi?

The interfaces are similar. The system I am putting together is pretty 
symmetrical and I like the language which is more consistent with 
twisted's use of factories. You were using this language before in your 
initial proposal to ngi but many of the interfaces are now name 
differently named. See how nice and symmetrical this looks :-)

clientconnfactoryhandler
clientconnhandler
clientconnfactory
clientreconnfactory
clientmethods
clientstorage
connection
serverconnfactory
serverconnhandler
serverconnfactoryhandler
servermethods
serverstorage

> 
> Note that I wrote ngi based on a misunderstanding of Twisted. :(  I 
> didn't realize that Twisted application code didn't touch sockets and 
> therefore could be tested without using sockets, threads or 
> subprocesses.  IMO, this is Twisted's most important feature.  I 
> recently finished my first Twisted application and I was very pleased 
> with my ability to create sane tests for it, although it's test support 
> infrastructure could use improvement, which I plan to do.
> 
>> I have got a basic transport to use prospective broker and banana 
>> protocol. So a client and server but I have not yet put together the 
>> methods for interacting with the storage.
> 
> I plan to stick with the existing ZEO protocol, both for compatibility 
> and for performance reasons.

Banana protocol is efficient and extensible. It has its own 
producer/consumer so no need to roll you own system as you have for ZEO. 
  I think you will find banana an efficient serialization of objects.

As far as compatibility, I am not sure the past always needs to equal 
the future. It is probably the methods for zrpc that are important, not 
the transport - at least that is my thinking. This is the obvious 
downside of ZEO at the moment is that is too tied to asyncore. In fact, 
  I started by trying to replicate much of what was in ZEO - then I 
threw it away because something simpler I believe is possible.

If a new system can transport the objects and execute the methods, isn't 
that all it is supposed to do? On top of it, what if something better 
comes around - the methods that you are executing on both sides are the 
important part and the transport should be transparent. I think this 
could produce a much cleaner and understandable system.

On top of it, it doesn't upset anything with current zeo for users that 
currently use this. It could in fact be developed and run in parallel, 
not that you would want to run it that way :-). Another obvious 
advantage is that twisted project has tests for most of this already so 
that is also a big plus.

Out of curiosity, does perspective broker
> support one-way calls, messages sent without replies?  Scanning the 
> docs, I can't tell.  Of course, not waiting for replies is inherent in 
> Twisted's defered model I suppose.

Yes

To set up something to work with I create both a TCP and SSL 
clientfactory so I bring them into the multiservice when zope is 
started. This way you do not need a separate loop to run the service 
which is the current way zeo runs. At the same time this does not 
interfere with the current zope setup. I have no proof one way or the 
other that this will not work as efficiently on a single twisted loop 
with the second asyncore loop. I still have to hook up database methods 
and I had put this on a backburner for a while since I have had other 
things to do.

In any case, the client setup in zope requires only minor modification 
of the zope.app.twisted main.py since the script only deals with servers 
and not clients at the moment. The clients are reconnecting clients so 
will continue to try reconnecting if connection is lost in gradually 
increasing time intervals.

On the server side I have the server working using zconfig and zdaemon 
in a similar way to way it is setup for zope itself. I have been looking 
at the zope3recipes since I am working on getting this into a buildout 
to continue since I am quite happy with the way buildouts work.

Regards,
David