[ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2

[Alan Runyan]

> How does data get into the ZEO storage then?

We have 10 ZEO clients that are for public consumption "READ ONLY".
We have a separate ZEO client that is writing that is on a separate box.

The website got slashdoted this morning and we had 4 zeo clients go
out.  Basically waiting for the zeo server for many minutes (basically
hung and reporting 500s back to browsers).

4/10/07 - delta ah2 lock up - http://paste.plone.org/13919
4/10/07 - epsilon ah2 lock up - http://paste.plone.org/13920
4/10/07 - delta ah1 lock up - http://paste.plone.org/13936
4/10/07 - epsilon ah3 lock up - http://paste.plone.org/13935

Just FYI: Varnish didnt go over 3% CPU during the traffic surge; over
200 req/second.

The ZEO Server was consistently at 2% CPU.  Lots of traffic was
getting through the cache and back to pound which was then load
balancing to the ZEO clients.

For Jim: We did not adjust the transaction timeout.  Would that have
helped in the case of READ's?

The customer was posting content throughout the slashdot.  The problem
was that when the clients would update they would end up through 500s

I am seeing something *very* strange in zeo.log:
2007-04-10T12:20:53 ERROR ZEO.zrpc.Connection(S) (172.16.235.120:49351) Erro
r caught in asyncore
Traceback (most recent call last):
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 391, in han
dle_read_event
    self.handle_read()
  File "/usr/local/zope/lib/python/ZEO/zrpc/smac.py", line 147, in handle_re
ad
    d = self.recv(8192)
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 343, in rec
v
    data = self.socket.recv(buffer_size)
error: (113, 'No route to host')


but several more of these:
2007-04-10T13:55:36 ERROR ZEO.zrpc.Connection(S) (172.16.235.119:44322) Erro
r caught in asyncore
Traceback (most recent call last):
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 69, in read
    obj.handle_read_event()
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 391, in han
dle_read_event
    self.handle_read()
  File "/usr/local/zope/lib/python/ZEO/zrpc/smac.py", line 147, in handle_re
ad
    d = self.recv(8192)
  File "/usr/local/python-2.4.4/lib/python2.4/asyncore.py", line 343, in rec
v
    data = self.socket.recv(buffer_size)
error: (110, 'Connection timed out')
------