[ZODB-Dev] Practical experience - ZEO-Server & limited size of file descriptors

[Andreas Gabriel]

Hello,

we are hosting a big ZEO based zope site and encountered a problem with a
limited size of file descriptors (FDs) of the ZEO server.

Scenario

  36 ZEO clients (each 2 threads)
  1  ZEO server  (serving 10 storages)

It was not possible to connect all ZEO clients to the ZEO server.
After a short amount of time following events occur in the event.log
of the ZEO clients

[snip]

2010-02-08T14:03:25 PROBLEM(100) zrpc:21615 CW: error connecting to
('zeo-server.dummy.de', 10000): ECONNREFUSED

[snip]

and simultaneously the ZEO server hangs and the whole site goes down.
Unfortunately, there was no hint in the ZEO server logs. After 'Googling'
we found the following hint

  http://comments.gmane.org/gmane.comp.web.zope.plone.user/101892

that each zeo client connection is consuming three file descriptors at the ZEO
server side. It was possible to calculate the theoretical required number of FDs
with this info

   75 (base) + 36 (zeo-clients) x 10 (storages) = 1155

We tried to open as many connections as possible to the ZEO server with a simple
script (see attachment) and counted the number of open FDs of the ZEO server using "lsof".
The result was that the ZEO server hangs at 1025 open FDs. Therefore, we assumed that
the OS (here Linux) limits the available number of FDs to 1024 by configuration.
Using "ulimit" (hard/soft) we increased the number of allowed open FDs to 2048.
However, there was no chance to open more than 329 (instead of 360) connections
(=1025 FDs) to the ZEO server :(

After looking at the sources, ZEO server uses the asyncore library to manage the
incoming connections. After *intensive* 'Googling' we have to notice that python's
asyncore library has a hard compiled in size limit of open FDs (namely 1024). The
limit is defined as macro __FD_SETSIZE in the header file of the libc6 library

    /usr/include/bits/typesizes.h

Therefore, it was unfortunately necessary to change the limit in the header file
to

  #define __FD_SETSIZE 2048

and to re-compile python's sources to overcome the problem. However, our ZEO scenario
now works with the re-compiled python interpreter :)

I hope you will find this information useful.
Kind regards
Andreas


-- 
Dr. Andreas Gabriel, Hochschulrechenzentrum, http://www.uni-marburg.de/hrz
Hans-Meerwein-Str., 35032 Marburg,  fon +49 (0)6421 28-23560  fax 28-26994
-------------------- Philipps-Universitaet Marburg -----------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: zeo-check-max-connections.py
Type: text/x-python
Size: 1978 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zodb-dev/attachments/20100719/786a0230/attachment.py