[ZODB-Dev] ZeoServer, multiple storages and open file handles

[Tim Godfrey]

Hi

Thanks for your responses thus far.

For the most part the zeoserver is performing quite well. Very low memory
and CPU usage.

I really think my problem lies in the fact that the Zope clients are
reporting these sort of messages quite regularly when communicating to ZEO:

2012-10-19T10:15:02 INFO ZEO.ClientStorage storage_d06_zeostorage Testing
connection <ManagedClientConnection ('10.221.0.247', 8101)>
------
2012-10-19T10:15:12 ERROR ZEO.zrpc (8356) CW: error in testConnection
(('10.221.0.247', 8101))
Traceback (most recent call last):
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py",
line 595, in test_connection
    self.preferred = self.client.testConnection(self.conn)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py",
line 565, in testConnection
    stub = self.StorageServerStubClass(conn)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py",
line 386, in stub
    raise ValueError("Timeout waiting for protocol handshake")
ValueError: Timeout waiting for protocol handshake

Each time that happens, more open file descriptors are used by zeo and as
far as I'm aware are never released unless the Zope client shuts down.

I have upped zeo's invalidation-queue-size to 4000 (not sure what my upper
limit is there) to see if that reduces the growth of open file handles. So
far that seems to be the case. 16 Zope clients connected with only 413 file
descriptors in use.

Not sure if any of these observations are related however?

Tim

On 19 October 2012 06:29, Jim Fulton <jim at zope.com> wrote:

> On Thu, Oct 18, 2012 at 3:09 PM, Leonardo Rochael Almeida
> <leorochael at gmail.com> wrote:
>
> Thanks for pitching in with an answer!
>
> > Having a single ZEO server handling more than one or two storages is
> > not usually a good idea. ZEO does not handle each storage in a
> > separate thread, so you're underusing multiple CPUs if you have them.
>
> Nit pick: ZEO handles each client connection and storage in a separate
> thread.
> (So 30 storages and 16 clients means 480 threads :)
>
> It is Python's GIL that prevents effective use of multiple processors.
>
> ZEO goes out if it's way to let I/O C code run concurrently (since I/O
> isn't
> subject to the GIL) and I've seen ZEO storage servers use up to 200% CPU
> on 4-core boxes (2 cores worth, IOW).
>
> Jim
>
> --
> Jim Fulton
> http://www.linkedin.com/in/jimfulton
> Jerky is better than bacon! http://zo.pe/Kqm
>



-- 
Tim Godfrey
Obsidian Consulting Group

P: +61 3 9355 7844
F: +61 3 9350 4097
E: tim at obsidian.com.au
W: http://www.obsidian.com.au/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20121019/c7e916fc/attachment.html>