[Zope3-dev] Tests hanging due to ZEO
Chris Withers
chrisw@nipltd.com
Wed, 26 Feb 2003 16:04:24 +0000
Jeremy Hylton wrote:
> I really can't help debug the problem if you don't tell me what exactly is
> happening.
It's pretty difficult to find out, hence I've been throwing out information as
and when I get it...
> We've run these tests on RH Linux 7.3 and Windows 2k, and didn't
> see any hangs there.
These are running on:
chrisw@seshat:~/ZopeTests/sandbox/Zope3$ uname -a
Linux seshat 2.4.20 #1 Thu Dec 19 10:10:15 GMT 2002 i686 unknown
I'm pretty sure it's a Debian box...
> It's not completely surprising that they hang -- lots
> of processes getting created and waiting for each other -- but we need to
> know more about what's going wrong to debug it.
Indeed... here's what I found so far:
http://mail.zope.org/pipermail/zope-coders/2003-February/003385.html
http://mail.zope.org/pipermail/zope-coders/2003-February/003398.html
...and...
> A good start would be to run the ZEO tests in isolation, in verbose mode:
> python test.py -bvv zeo
It didn't hang, but I got the following errors:
======================================================================
ERROR: checkNoVerificationOnServerRestart (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 66, in setUp
self.startServer()
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkNoVerificationOnServerRestartWith2Clients (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 66, in setUp
self.startServer()
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkQuickVerificationWith2Clients (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 66, in setUp
self.startServer()
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReadOnlyFallbackReadOnlyStorage (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 66, in setUp
self.startServer()
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReconnectSwitch (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 568, in checkReconnectSwitch
self.startServer(index=1, read_only=False)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
Now, given what Guido said in reply to my email, and the fact that the Zope 3
tests only started hanging last night, we should have a bit of a smaller set of
stuff to hunt through (merges from stuff that would have appeared in Zope 2's
HEAD yesterday to stuff that would have appeared in Zope 3's HEAD last night...).
Anyway, I ran the tests the same way the autotest does:
chrisw@seshat:~/ZopeTests/sandbox/Zope3$ python2.2 test.py
Running tests from /stuff/chrisw/ZopeTests/sandbox/Zope3/build
Failure in test checkNoVerificationOnServerRestart (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 593, in checkNoVerificationOnServerRestart
self.assertEqual(self._storage.verify_result, "no verification")
File "/stuff/usr/local/python2.2/Lib/unittest.py", line 286, in failUnlessEqual
raise self.failureException, \
AssertionError: None != 'no verification'
Error in test checkReadOnlyFallbackReadOnlyStorage (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 468, in checkReadOnlyFallbackReadOnlyStorage
self.startServer(create=False, index=0, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
Error in test checkReadOnlyStorage (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 455, in checkReadOnlyStorage
self.startServer(create=False, index=0, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
Error in test checkReconnectFallback (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 516, in checkReconnectFallback
self.startServer(create=False, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
Error in test checkReconnectSwitch (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 568, in checkReconnectSwitch
self.startServer(index=1, read_only=False)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReadOnlyFallbackReadOnlyStorage (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 468, in checkReadOnlyFallbackReadOnlyStorage
self.startServer(create=False, index=0, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReadOnlyStorage (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 455, in checkReadOnlyStorage
self.startServer(create=False, index=0, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReconnectFallback (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 516, in checkReconnectFallback
self.startServer(create=False, read_only=True)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
ERROR: checkReconnectSwitch (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 568, in checkReconnectSwitch
self.startServer(index=1, read_only=False)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 129, in startServer
self.monitor, self.keep, self.invq, self.timeout)
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/forker.py",
line 102, in start_zeo_server
s.connect(adminaddr)
error: (111, 'Connection refused')
======================================================================
FAIL: checkNoVerificationOnServerRestart (<class
'zodb.zeo.tests.test_conn.FileStorageConfig:ReconnectionTests'>)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/stuff/chrisw/ZopeTests/sandbox/Zope3/build/lib.linux-i686-2.2/zodb/zeo/tests/connection.py",
line 593, in checkNoVerificationOnServerRestart
self.assertEqual(self._storage.verify_result, "no verification")
File "/stuff/usr/local/python2.2/Lib/unittest.py", line 286, in failUnlessEqual
raise self.failureException, \
AssertionError: None != 'no verification'
----------------------------------------------------------------------
Ran 3491 tests in 485.837s
Strangely, no hang. I wodner if someone's done something to fix it today?
Anyway, let me know how I can help more and when you think I should give the
Zope 3 tests another go...
cheers,
Chris