[ZODB-Dev] Practical experience - ZEO-Server & limited size of file descriptors

[Jim Fulton]

On Mon, Jul 19, 2010 at 9:40 AM, Andreas Gabriel
<gabriel at hrz.uni-marburg.de> wrote:
> Hello,
>
> we are hosting a big ZEO based zope site and encountered a problem with a
> limited size of file descriptors (FDs) of the ZEO server.
>
> Scenario
>
>  36 ZEO clients (each 2 threads)
>  1  ZEO server  (serving 10 storages)

I advise against serving more than one storage per server.

> It was not possible to connect all ZEO clients to the ZEO server.
> After a short amount of time following events occur in the event.log
> of the ZEO clients
>
> [snip]
>
> 2010-02-08T14:03:25 PROBLEM(100) zrpc:21615 CW: error connecting to
> ('zeo-server.dummy.de', 10000): ECONNREFUSED
>
> [snip]
>
> and simultaneously the ZEO server hangs and the whole site goes down.
> Unfortunately, there was no hint in the ZEO server logs. After 'Googling'
> we found the following hint
>
>  http://comments.gmane.org/gmane.comp.web.zope.plone.user/101892
>
> that each zeo client connection is consuming three file descriptors at the ZEO
> server side. It was possible to calculate the theoretical required number of FDs
> with this info
>
>   75 (base) + 36 (zeo-clients) x 10 (storages) = 1155
>
> We tried to open as many connections as possible to the ZEO server with a simple
> script (see attachment) and counted the number of open FDs of the ZEO server using "lsof".
> The result was that the ZEO server hangs at 1025 open FDs. Therefore, we assumed that
> the OS (here Linux) limits the available number of FDs to 1024 by configuration.
> Using "ulimit" (hard/soft) we increased the number of allowed open FDs to 2048.
> However, there was no chance to open more than 329 (instead of 360) connections
> (=1025 FDs) to the ZEO server :(
>
> After looking at the sources, ZEO server uses the asyncore library to manage the
> incoming connections. After *intensive* 'Googling' we have to notice that python's
> asyncore library has a hard compiled in size limit of open FDs (namely 1024). The
> limit is defined as macro __FD_SETSIZE in the header file of the libc6 library
>
>    /usr/include/bits/typesizes.h
>
> Therefore, it was unfortunately necessary to change the limit in the header file
> to
>
>  #define __FD_SETSIZE 2048
>
> and to re-compile python's sources to overcome the problem. However, our ZEO scenario
> now works with the re-compiled python interpreter :)

A simpler solution is to run each storage in a separate process.

(In ZODB 3.9, only one fd is needed per client.  ZODB 3.10 goes back
 to requiring 3.)

Jim

--
Jim Fulton