[ZODB-Dev] what could cause slowness checking if databases are alive?

[Chris Withers]

Hi All,

This is all on Zope 2.9.8 for both the ZEO server and the app server 
clients.

I have a customer with 2 app servers behind a hardware loadbalancer. The 
load balancer uses a hash of a url on each app server to determine if 
the app server is alive. Likewise, if the app server doesn't respond to 
that status url, or takes too long to respond, the load balancer drops 
the app server out of the pool.

Each app server has 1 zope app process that's proxied through an apache 
instance.

The status page is generated by the following external method:

import socket
from ZEO.ClientStorage import ClientStorage
template = """
This is app server: %s
Connected as follows:
%s
"""
def status(self):
     connections = []
     databases = self.Control_Panel.Database
     for name in sorted(databases.getDatabaseNames()):
         storage = databases[name]._getDB()._storage
         if not isinstance(storage,ClientStorage):
             continue
         connections.append('%s: %r'%(name,storage._addr))
     return template % (
         socket.gethostname(),
         '\n'.join(connections)
         )

...so as to check that the ZEO server connections are all fine. However, 
the code that prints the storage name and address seems to cause slow 
responses every so often. Sadly, they're so slow the load balancer drops 
the app server out of the pool. This is bad when it happens to both app 
servers :-S

We added logging to the apache instances to show how long it took to 
serve the status requests. It looks like the status url normally takes 
around 3-4ms to serve, however some requests are taking longer (1.7s and 
even up to 70s on occasion)

Any idea what might be causing this slowness, or how I can find out what 
might be causing this slowness?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk