[ZODB-Dev] Zope or Zeo hanging up sockets

[Matt]

I am getting a really strange behavior with requests with Zope on ZEO.  Not
sure what part of the equation is to blame.  Our site's busy time consists
of about 20-40 requests per second, mainly because we have some heavy caches
in front of us.  Sometimes we are finding that the Zope process suddenly
accelerates to 99.9% CPU and stays there.  A sockstat shows that connections
are piling up to Zope.  I watched the logs for a while and found that when a
certain external method was called that takes up to 10 seconds to run and
return, that all the other requests pile up behind it.  When it finally goes
through then a burst of some, but not all, of other pending requests goes
through, I am guessing as many as can fit before the method has been called
again.  To me this seems as if there is only one thread of python processing
these requests, though in other instances I know for sure that a longer
process, on the order of minutes, is being processed by Zope, and I can
still see requests coming through.  I was wondering where to turn to for
debugging.  The following is my setup :

At present I am running ZEO on a UNIX socket as I only have one Zope looking
at it right now, but not for long.  I have also switched back to Data.fs
storage to get the speed increase, but was experiencing the same socket
lockup while using bdb storage.

Zope Version (Zope 2.4.0 (source release, python 2.1, linux2), python 2.1.0,
freebsd4)
Python Version 2.1 (#1, Jul 18 2001, 17:06:28) [GCC 2.95.3 [FreeBSD]
20010315 (release)]
System Platform freebsd4

the guts of me start scripts are as follows :

Zope :

exec chroot /usr/local/sandbox /usr/local/bin/python \
    $INST_HOME/z2.py \
    -m '' -f "${FTPPORT}" -a ${MYIP} -p '' -w "${WWWPORT}" -F '' \
    -Z ''

Zeo :

exec chroot /usr/local/sandbox /usr/local/bin/python \
    $INST_HOME/lib/python/ZEO/start.py -U $INST_HOME/var/zeo.soc \
    -S main=StorageConfig:data_storage \
    -S bdb=StorageConfig:bdb_storage \
    ZEO_SERVER_PID=$INST_HOME/var/ZEO_SERVER.pid \
    -s "$@"

regards
Matt