[ZODB-Dev] ZEO server crash

[Chris McDonough]

FWIW, the shell also limits the number of open files that each user can
have open.  The default is typically a puny 1024 files (you can check
this by running ulimit -n as the user under which you're running the ZEO
server).  To bump this up, under Linux you typically need to set a line
like this in your /etc/security/limits.conf file and restart the system:

* soft nofile 65535
* hard nofile 65535

Obviously if one of your apps (ZEO or another app that happens to run as
this user under the same parent shell) is leaking file descriptors, this
is only a stopgap.  You can probably do "watch cat /proc/sys/fs/file-nr"
to see how the number of filedescriptors grows; maybe you can get a
sense of what's causing it by interactive inspection. 

- C



On Thu, 2005-03-31 at 18:34, Tim Peters wrote:
> [4mi(r|x)]
> > I am using ZEO as a server on a Debian linux 2.6 machine. The ZEO version
> > is 3.3.
> >
> > I get following error:
> > error: uncaptured python exception, closing channel
> > <ZEO.zrpc.server.Dispatcher listening 192.168.1.100:10030 at
>   0xb79ec56c> (exceptions.OSError:[Errno 24] Too many open files
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal
> lback.py|poll|129]
> > [/usr/lib/python2.3/asyncore.py|handle_read_event|384] >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/server.py|ha
> ndle_accept|58]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/StorageServer.py|
> new_connection|803]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p
> y|__init__|566]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p
> y|__init__|159]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p
> y|_prepare_async|421]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ThreadedAsync/LoopCal
> lback.py|register_loop_callback|64]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/connection.p
> y|set_async|426]
> >
> [/home/amix/Desktop/ZODB3-3.3/build/lib.linux-i686-2.3/ZEO/zrpc/trigger.py|_
> _init__|55])
> >
> > I have tried to set the /proc/sys/fs/files-max up but that did not help
> > :-(
> 
> Meaning what?  You get exactly the same traceback every time?  Every time
> you try to connect to ZEO?  Once an hour?  Once a day?  Too much telepathy
> required here.  Are you using Zope?  Is this a standalone application?  Are
> other apps on the box mucking with a large number of files?  Etc.
> 
> > Is there any fix for this?
> 
> I don't know.  The only other report I found was from 2001, and didn't
> appear to be resolved at the time:
> 
>     http://mail.zope.org/pipermail/zodb-dev/2001-October/001616.html
> 
> Since others don't report "Too many open files" errors, presumption has to
> favor that you're doing something unusual, but I can't guess what.  Line 55
> in trigger.py is
> 
>             r, w = self._fds = os.pipe()
> 
> so all we can tell from the traceback is that the OS ran out of file
> descriptors and so can't create a new pipe.  The descriptors opened there
> are closed again by the trigger.close() method, so I suppose that if the
> latter isn't getting called, open descriptors would build up.  
> 
> _______________________________________________
> For more information about ZODB, see the ZODB Wiki:
> http://www.zope.org/Wikis/ZODB/
> 
> ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> http://mail.zope.org/mailman/listinfo/zodb-dev
>