[ZODB-Dev] ZeoServer, multiple storages and open file handles

[Leonardo Rochael Almeida]

Having a single ZEO server handling more than one or two storages is
not usually a good idea. ZEO does not handle each storage in a
separate thread, so you're underusing multiple CPUs if you have them.

Multiple ZEO servers would also spread the open sockets among them.
There could still be a leak, but it'd take longer to exhaust file
descriptors.

On Wed, Oct 17, 2012 at 11:29 PM, Tim Godfrey <tim at obsidian.com.au> wrote:
> Hi all
>
> We have a customer on Plone 4.1 and recently we started to see a great
> increase in the number of open file descriptors the Zeoserver process uses.
>
> We have a pretty big system distributed over many servers as the Plone
> system gets a lot of requests. Currently we are running a Zeoserver with 30
> storages and have 16 Zope clients that connect to it.
>
> I've had Zeoserver in debug mode to try and get some more insight into the
> problem and I'm seeing a lot of these errors:
>
> Exception in thread Thread-575:
> Traceback (most recent call last):
>   File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in
> __bootstrap_inner
>     self.run()
>   File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in
> run
>     self.__target(*self.__args, **self.__kwargs)
>   File
> "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py",
> line 661, in server_loop
>     asyncore.poll(30.0, map)
>   File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in
> poll
>     r, w, e = select.select(r, w, e, timeout)
> ValueError: filedescriptor out of range in select()
>
> Exception in thread Thread-576:
> Traceback (most recent call last):
>   File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in
> __bootstrap_inner
>     self.run()
>   File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in
> run
>     self.__target(*self.__args, **self.__kwargs)
>   File
> "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py",
> line 661, in server_loop
>     asyncore.poll(30.0, map)
>   File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in
> poll
>     r, w, e = select.select(r, w, e, timeout)
> ValueError: filedescriptor out of range in select()
>
> There's also a lot of these lines in zeoserver.log:
>
> 2012-10-18T12:42:08 new connection ('10.221.0.144', 34919):
> <ManagedServerConnection ('10.221.0.144', 34919)>
> 2012-10-18T12:42:08 new connection ('10.221.0.64', 41539):
> <ManagedServerConnection ('10.221.0.64', 41539)>
>
> Each time a new connection is logged further open file descriptors are used.
>
> There are also these errors in the Zope clients:
>
> 2012-10-18T13:20:01 ERROR ZEO.zrpc (7746) CW: error in testConnection
> (('10.221.0.247', 8101))
> Traceback (most recent call last):
>   File
> "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py",
> line 595, in test_connection
>     self.preferred = self.client.testConnection(self.conn)
>   File
> "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py",
> line 565, in testConnection
>     stub = self.StorageServerStubClass(conn)
>   File
> "/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py",
> line 386, in stub
>     raise ValueError("Timeout waiting for protocol handshake")
> ValueError: Timeout waiting for protocol handshake
>
> At time of writing the Zeoserver process is using 1604 file descriptors but
> that number will continue to grow. I suppose I'm wondering why it's always
> creating new fds and not closing old ones. Also whether there are changes I
> can make to my configuration to stop these errors and growing number of file
> descriptors.
>
> If Zeoserver is allowed to keep running it will eventually hit the operating
> system limit of open fds and start raising "Too Many Open File" Exceptions
> which causes havoc with transactions are I'm sure you would know.
>
> Is anyone able to offer some advice? I'm more than happy to provide more
> information if required.
>
> Thanks,
>
> Tim
>
>
> _______________________________________________
> For more information about ZODB, see http://zodb.org/
>
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> https://mail.zope.org/mailman/listinfo/zodb-dev
>