[ZODB-Dev] ZeoServer, multiple storages and open file handles

[Tim Godfrey]

Hi all

We have a customer on Plone 4.1 and recently we started to see a great
increase in the number of open file descriptors the Zeoserver process uses.

We have a pretty big system distributed over many servers as the Plone
system gets a lot of requests. Currently we are running a Zeoserver with 30
storages and have 16 Zope clients that connect to it.

I've had Zeoserver in debug mode to try and get some more insight into the
problem and I'm seeing a lot of these errors:

Exception in thread Thread-575:
Traceback (most recent call last):
  File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
    self.run()
  File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in
run
    self.__target(*self.__args, **self.__kwargs)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py",
line 661, in server_loop
    asyncore.poll(30.0, map)
  File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in
poll
    r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()

Exception in thread Thread-576:
Traceback (most recent call last):
  File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 532, in
__bootstrap_inner
    self.run()
  File "/opt/informa/Python-2.6/lib/python2.6/threading.py", line 484, in
run
    self.__target(*self.__args, **self.__kwargs)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/connection.py",
line 661, in server_loop
    asyncore.poll(30.0, map)
  File "/opt/informa/Python-2.6/lib/python2.6/asyncore.py", line 140, in
poll
    r, w, e = select.select(r, w, e, timeout)
ValueError: filedescriptor out of range in select()

There's also a lot of these lines in zeoserver.log:

2012-10-18T12:42:08 new connection ('10.221.0.144', 34919):
<ManagedServerConnection ('10.221.0.144', 34919)>
2012-10-18T12:42:08 new connection ('10.221.0.64', 41539):
<ManagedServerConnection ('10.221.0.64', 41539)>

Each time a new connection is logged further open file descriptors are used.

There are also these errors in the Zope clients:

2012-10-18T13:20:01 ERROR ZEO.zrpc (7746) CW: error in testConnection
(('10.221.0.247', 8101))
Traceback (most recent call last):
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/zrpc/client.py",
line 595, in test_connection
    self.preferred = self.client.testConnection(self.conn)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ClientStorage.py",
line 565, in testConnection
    stub = self.StorageServerStubClass(conn)
  File
"/opt/informa/buildout-cache/eggs/ZODB3-3.10.3-py2.6-linux-x86_64.egg/ZEO/ServerStub.py",
line 386, in stub
    raise ValueError("Timeout waiting for protocol handshake")
ValueError: Timeout waiting for protocol handshake

At time of writing the Zeoserver process is using 1604 file descriptors but
that number will continue to grow. I suppose I'm wondering why it's always
creating new fds and not closing old ones. Also whether there are changes I
can make to my configuration to stop these errors and growing number of
file descriptors.

If Zeoserver is allowed to keep running it will eventually hit the
operating system limit of open fds and start raising "Too Many Open File"
Exceptions which causes havoc with transactions are I'm sure you would know.

Is anyone able to offer some advice? I'm more than happy to provide more
information if required.

Thanks,

Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zodb-dev/attachments/20121018/91a34265/attachment.html>