From klm at cvs.zope.org Tue Aug 7 10:55:56 2001 From: klm at cvs.zope.org (Ken Manheimer) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO - postcommit_actions:1.1 Message-ID: <200108071455.KAA32608@serenade.digicool.com> Update of /newrepo/ZEO In directory serenade:/home/klm/checkouts/newrepo/newint/ZEO Added Files: postcommit_actions Log Message: === Added File ZEO/postcommit_actions === From klm at zope.com Tue Aug 7 11:01:51 2001 From: klm at zope.com (Ken Manheimer) Date: Sun Aug 10 16:31:16 2008 Subject: Please ignore: [ZEO-Checkins] CVS: ZEO - postcommit_actions:1.1 In-Reply-To: <200108071455.KAA32608@serenade.digicool.com> Message-ID: Please ignore that message - i'm poking at the checkin notification machinery, and accidentally misdirected a message - the checkin didn't actually go in the ZEO section... Ken klm@zope.com On Tue, 7 Aug 2001, Ken Manheimer wrote: > Update of /newrepo/ZEO > In directory serenade:/home/klm/checkouts/newrepo/newint/ZEO > > Added Files: > postcommit_actions > Log Message: > > > > === Added File ZEO/postcommit_actions === > > > _______________________________________________ > ZEO-Checkins maillist - ZEO-Checkins@zope.org > http://lists.zope.org/mailman/listinfo/zeo-checkins > From jeremy at zope.com Tue Aug 7 13:24:23 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO - CHANGES.txt:1.25.2.3 Message-ID: <200108071724.f77HONq29725@cvs.baymountain.com> Update of /cvs-repository/ZEO In directory cvs.zope.org:/tmp/cvs-serv29714 Modified Files: Tag: zeo-1_0-branch CHANGES.txt Log Message: Add note about Jim's undo invalidation bug fix === ZEO/CHANGES.txt 1.25.2.2 => 1.25.2.3 === - ZEO 1.0 final + ZEO 1.0 beta 4 Bugs fixed + + - Fixed a bug in undo handling that caused connections to be reset + and prevented undos from propagating to all clients. - Fixed a bug that prevented a client from reconnecting to a server if the server restarted. From jeremy at zope.com Tue Aug 7 18:32:31 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - testZEO.py:1.9.2.1 Message-ID: <200108072232.f77MWVs11946@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv11925 Modified Files: Tag: zeo-1_0-branch testZEO.py Log Message: Variety of fixes that get testZEO to run successfully again Make ports for ZEO testing more random, reducing likelihood of port already in use errors. In tests for reconnection, catch ClientDisconnected, thread.error, and socket.error. XXX This is a bloody pain. We're placing a heavy burden on users to catch a plethora of exceptions in order to write robust code. Need to think about implementing John Heintz's suggestion to make sure all exceptions inherit from POSException. Add test_suite() method for conformance with Zope testing style. === StandaloneZODB/ZEO/tests/testZEO.py 1.9 => 1.9.2.1 === import os import random +import socket import sys import tempfile import time @@ -175,8 +176,7 @@ __super_setUp = StorageTestBase.StorageTestBase.setUp __super_tearDown = StorageTestBase.StorageTestBase.tearDown - ports = range(29000, 30000, 10) # enough for 100 tests - random.shuffle(ports) + ports = [random.randrange(25000, 30000) for i in range(200)] def setUp(self): """Start a ZEO server using a Unix domain socket @@ -186,7 +186,8 @@ """ self.running = 1 self.__fs_base = tempfile.mktemp() - self.addr = '', random.randrange(2000, 3000) + port = self.ports.pop() + self.addr = '', port pid, exit = self._startServer() self._pid = pid self._server = exit @@ -275,10 +276,16 @@ self._pid, self._server = self._startServer(create=0) oid = self._storage.new_oid() obj = MinPO(12) + # XXX This is a bloody pain. We're placing a heavy burden on + # users to catch a plethora of exceptions in order to write + # robust code. Need to think about implementing John Heintz's + # suggestion to make sure all exceptions inherit from + # POSException. + while 1: try: revid1 = self._dostore(oid, data=obj) - except (ClientDisconnected, thread.error), err: + except (ClientDisconnected, thread.error, socket.error), err: get_transaction().abort() time.sleep(0.1) else: @@ -304,6 +311,9 @@ if meth.startswith(name): suite.addTest(klass(meth)) return suite + +def test_suite(): + return makeTestSuite() def main(): import sys, getopt From jeremy at zope.com Tue Aug 7 18:32:53 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - forker.py:1.5.2.1 Message-ID: <200108072232.f77MWra11988@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv11967 Modified Files: Tag: zeo-1_0-branch forker.py Log Message: Try harder to avoid port already in use errors === StandaloneZODB/ZEO/tests/forker.py 1.5 => 1.5.2.1 === if domain == "AF_INET": import random - addr = '', random.randrange(2000, 3000) + addr = '', random.randrange(25000, 30000) elif domain == "AF_UNIX": import tempfile addr = tempfile.mktemp() else: raise ValueError, "bad domain: %s" % domain - pid, exit = start_zeo_server(storage, addr) + for i in 1, 2, 3: + try: + pid, exit = start_zeo_server(storage, addr) + except socket.error, (num, msg): + if num == 98: + continue + else: + raise + else: + break s = ZEO.ClientStorage.ClientStorage(addr, storage_id, debug=1, client=cache, cache_size=cache_size, From jeremy at zope.com Tue Aug 7 18:37:56 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientStorage.py:1.33.2.1 Message-ID: <200108072237.f77Mbu612467@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv12451/ZEO Modified Files: Tag: zeo-1_0-branch ClientStorage.py Log Message: XXX Provisional change Don't release the commit lock unless there is a transaction in progress. This appears to cause the "release unlocked lock" error to be replaced with a socket.error: "bad file descriptor". That's progress? === StandaloneZODB/ZEO/ClientStorage.py 1.33 => 1.33.2.1 === self._transaction=None thread.start_new_thread(self._call.connect,(0,)) - try: self._commit_lock_release() - except: pass + if self._transaction is not None: + try: + self._commit_lock_release() + except: + pass def becomeAsync(self, map): self._lock_acquire() @@ -479,6 +482,7 @@ "This action is temporarily unavailable.

") r=self._call(self.__begin, id, user, desc, ext) except: + # XXX can't seem to guarantee that the lock is held here. self._commit_lock_release() raise From jeremy at zope.com Wed Aug 8 12:25:49 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - trigger.py:1.1.8.1 Message-ID: <200108081625.f78GPnm24214@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv24198 Modified Files: Tag: zeo-1_0-branch trigger.py Log Message: Shorten two long lines, one by wrapping, another by removing an unnecessary tuple unpacking. === StandaloneZODB/ZEO/trigger.py 1.1 => 1.1.8.1 === thunk() except: - (file, fun, line), t, v, tbinfo = asyncore.compact_traceback() - print 'exception in trigger thunk: (%s:%s %s)' % (t, v, tbinfo) + nil, t, v, tbinfo = asyncore.compact_traceback() + print ('exception in trigger thunk:' + ' (%s:%s %s)' % (t, v, tbinfo)) self.thunks = [] finally: self.lock.release() From jeremy at zope.com Wed Aug 8 12:28:21 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - trigger.py:1.1.8.2 Message-ID: <200108081628.f78GSL224382@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv24366 Modified Files: Tag: zeo-1_0-branch trigger.py Log Message: Apply long-line fixes in win32 code branch. Add XXX comment about need to refactor to avoid code duplication. === StandaloneZODB/ZEO/trigger.py 1.1.8.1 => 1.1.8.2 === else: + # XXX Should define a base class that has the common methods and + # then put the platform-specific in a subclass named trigger. + # win32-safe version class trigger (asyncore.dispatcher): @@ -246,12 +249,9 @@ try: thunk() except: - (file, fun, line), t, v, tbinfo = asyncore.compact_traceback() - print 'exception in trigger thunk: (%s:%s %s)' % (t, v, tbinfo) + nil, t, v, tbinfo = asyncore.compact_traceback() + print ('exception in trigger thunk:' + ' (%s:%s %s)' % (t, v, tbinfo)) self.thunks = [] finally: self.lock.release() - - - - From jeremy at zope.com Wed Aug 8 13:31:39 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - StorageServer.py:1.28.2.2 Message-ID: <200108081731.f78HVdr27082@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv27064 Modified Files: Tag: zeo-1_0-branch StorageServer.py Log Message: Avoid long delays at the end of a pack. When a client calls pack, a separate thread is started to call the pack call on the storage. When the thread finishes, it calls message_output() to send a response to the client. If asyncore is currently in a poll call when this happens, the output won't be detected until the next poll call. If there is little I/O on the connection, this won't happen until the select call times out after 30 seconds. The trigger is the standard gimmick for one thread to notify a mainloop in another thread that the first thread has some I/O to do. It exits the current select. The next poll call detects that the triggering thread is ready to do I/O and handles it. === StandaloneZODB/ZEO/StorageServer.py 1.28.2.1 => 1.28.2.2 === from ZODB.Transaction import Transaction import traceback -from zLOG import LOG, INFO, ERROR, TRACE +from zLOG import LOG, INFO, ERROR, TRACE, BLATHER from ZODB.referencesf import referencesf from thread import start_new_thread from cStringIO import StringIO +from ZEO import trigger class StorageServerError(POSException.StorageError): pass @@ -124,7 +125,6 @@ self.__connections={} self.__get_connections=self.__connections.get - asyncore.dispatcher.__init__(self) if type(connection) is type(''): @@ -234,6 +234,7 @@ self.__server=server self.__invalidated=[] self.__closed=None + self._pack_trigger = trigger.trigger() if __debug__: debug='ZEO Server' else: debug=0 SizedMessageAsyncConnection.__init__(self, sock, addr, debug=debug) @@ -384,16 +385,21 @@ if wait: return _noreturn def _pack(self, t, wait=0): - try: + try: + LOG('ZEO Server', BLATHER, 'pack begin') self.__storage.pack(t, referencesf) + LOG('ZEO Server', BLATHER, 'pack end') except: LOG('ZEO Server', ERROR, 'Pack failed for %s' % self.__storage_id, error=sys.exc_info()) - if wait: self.return_error(sys.exc_info()[0], sys.exc_info()[1]) + if wait: + self.return_error(sys.exc_info()[0], sys.exc_info()[1]) + self._pack_trigger.pull_trigger() else: if wait: self.message_output('RN.') + self._pack_trigger.pull_trigger() else: # Broadcast new size statistics self.__server.invalidate(0, self.__storage_id, (), From jeremy at zope.com Thu Aug 9 18:14:10 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - zrpc.py:1.18.2.1 Message-ID: <200108092214.f79MEA423825@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv23809 Modified Files: Tag: zeo-1_0-branch zrpc.py Log Message: Add definition for EINTR === StandaloneZODB/ZEO/zrpc.py 1.18 => 1.18.2.1 === from zLOG import LOG, TRACE, DEBUG, INFO +from errno import EINTR + # We create a special fast pickler! This allows us # to create slightly more efficient pickles and # to create them a tad faster. From jeremy at zope.com Thu Aug 16 16:27:49 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientCache.py:1.17 Message-ID: <200108162027.f7GKRnq13006@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv12945 Modified Files: ClientCache.py Log Message: Robustification-- catch OSError when tempfile is closed. === StandaloneZODB/ZEO/ClientCache.py 1.16 => 1.17 === def close(self): - self._f[self._current].close() + try: + self._f[self._current].close() + except OSError: + pass def open(self): self._acquire() From jeremy at zope.com Thu Aug 16 16:32:20 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - winserver.py:1.1 forker.py:1.6 testZEO.py:1.10 Message-ID: <200108162032.f7GKWKm13505@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv13314 Modified Files: forker.py testZEO.py Added Files: winserver.py Log Message: Add rudiments of a ZEO test framework for Windows winserver.py: A small script that runs a ZEO StorageServer and a ZEOTestServer. The ZEOTestServer is used to kill the StorageServer during a unit test's teardown phases. As soon as a client connects to that server, it exits the process. forker.py: Now forks and spawns, depending on os. XXX This needs to be cleaned up a lot, since the interfaces on Windows and Unix are nothing alike. testZEO.py: Add WindowsGenericTests and WindowsZEOFileStorageTests. These test cases use the Windows-specific forker interface. They also use getStorageInfo() to describe the storage, rather than getStorage() to create the storage. This is necessary because the storage instance is created in a separate process. === Added File StandaloneZODB/ZEO/tests/winserver.py === """Helper file used to launch ZEO server for Windows tests""" import asyncore import os import random import socket import threading import types import ZEO.StorageServer class ZEOTestServer(asyncore.dispatcher): """A trivial server for killing a server at the end of a test The server calls os._exit() as soon as it is connected to. No chance to even send some data down the socket. """ __super_init = asyncore.dispatcher.__init__ def __init__(self, addr): self.__super_init() if type(addr) == types.StringType: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) else: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.bind(addr) self.listen(5) def handle_accept(self): sock, addr = self.accept() os._exit(0) def load_storage_class(name): package = __import__("ZODB." + name) mod = getattr(package, name) return getattr(mod, name) def main(port, storage_name, args): klass = load_storage_class(storage_name) storage = klass(*args) zeo_port = int(port) test_port = zeo_port + 1 t = ZEOTestServer(('', test_port)) ## t = threading.Thread(target=ZEOTestServer, args=(('', test_port),)) ## t.start() serv = ZEO.StorageServer.StorageServer(('', zeo_port), {'1': storage}) asyncore.loop() if __name__ == "__main__": import sys main(sys.argv[1], sys.argv[2], sys.argv[3:]) === StandaloneZODB/ZEO/tests/forker.py 1.5 => 1.6 === import os import profile +import random +import socket import sys +import threading import time import types import ThreadedAsync @@ -12,80 +15,98 @@ PROFILE = 0 -class ZEOServerExit(asyncore.file_dispatcher): - """Used to exit ZEO.StorageServer when run is done""" +if os.name == "nt": - def writable(self): - return 0 + def start_zeo_server(storage_name, args): + """Start a ZEO server in a separate process. - def readable(self): - return 1 + Returns the ZEO port, the test server port, and the pid. + """ + import ZEO.tests.winserver + port = random.randrange(20000, 30000) + script = ZEO.tests.winserver.__file__ + if script.endswith('.pyc'): + script = script[:-1] + args = (sys.executable, script, str(port), storage_name) + args + pid = os.spawnv(os.P_NOWAIT, sys.executable, args) + return ('localhost', port), ('localhost', port + 1), pid + +else: + + class ZEOServerExit(asyncore.file_dispatcher): + """Used to exit ZEO.StorageServer when run is done""" + + def writable(self): + return 0 + + def readable(self): + return 1 + + def handle_read(self): + buf = self.recv(4) + if buf: + assert buf == "done" + asyncore.socket_map.clear() - def handle_read(self): - buf = self.recv(4) - if buf: - assert buf == "done" + def handle_close(self): asyncore.socket_map.clear() - - def handle_close(self): - asyncore.socket_map.clear() - -class ZEOClientExit: - """Used by client to cause server to exit""" - def __init__(self, pipe): - self.pipe = pipe - - def close(self): - os.write(self.pipe, "done") - -def start_zeo_server(storage, addr): - rd, wr = os.pipe() - pid = os.fork() - if pid == 0: - if PROFILE: - p = profile.Profile() - p.runctx("run_server(storage, addr, rd, wr)", globals(), - locals()) - p.dump_stats("stats.s.%d" % os.getpid()) + + class ZEOClientExit: + """Used by client to cause server to exit""" + def __init__(self, pipe): + self.pipe = pipe + + def close(self): + os.write(self.pipe, "done") + + def start_zeo_server(storage, addr): + rd, wr = os.pipe() + pid = os.fork() + if pid == 0: + if PROFILE: + p = profile.Profile() + p.runctx("run_server(storage, addr, rd, wr)", globals(), + locals()) + p.dump_stats("stats.s.%d" % os.getpid()) + else: + run_server(storage, addr, rd, wr) + os._exit(0) + else: + os.close(rd) + return pid, ZEOClientExit(wr) + + def run_server(storage, addr, rd, wr): + # in the child, run the storage server + os.close(wr) + ZEOServerExit(rd) + serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) + asyncore.loop() + storage.close() + if isinstance(addr, types.StringType): + os.unlink(addr) + + def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", + storage_id="1", cache_size=20000000): + """Setup ZEO client-server for storage. + + Returns a ClientStorage instance and a ZEOClientExit instance. + + XXX Don't know if os.pipe() will work on Windows. + """ + + if domain == "AF_INET": + import random + addr = '', random.randrange(2000, 3000) + elif domain == "AF_UNIX": + import tempfile + addr = tempfile.mktemp() else: - run_server(storage, addr, rd, wr) - os._exit(0) - else: - os.close(rd) - return pid, ZEOClientExit(wr) - -def run_server(storage, addr, rd, wr): - # in the child, run the storage server - os.close(wr) - ZEOServerExit(rd) - serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) - asyncore.loop() - storage.close() - if isinstance(addr, types.StringType): - os.unlink(addr) - -def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", - storage_id="1", cache_size=20000000): - """Setup ZEO client-server for storage. - - Returns a ClientStorage instance and a ZEOClientExit instance. - - XXX Don't know if os.pipe() will work on Windows. - """ - - if domain == "AF_INET": - import random - addr = '', random.randrange(2000, 3000) - elif domain == "AF_UNIX": - import tempfile - addr = tempfile.mktemp() - else: - raise ValueError, "bad domain: %s" % domain - - pid, exit = start_zeo_server(storage, addr) - s = ZEO.ClientStorage.ClientStorage(addr, storage_id, - debug=1, client=cache, - cache_size=cache_size, - min_disconnect_poll=0.5) - return s, exit, pid + raise ValueError, "bad domain: %s" % domain + + pid, exit = start_zeo_server(storage, addr) + s = ZEO.ClientStorage.ClientStorage(addr, storage_id, + debug=1, client=cache, + cache_size=cache_size, + min_disconnect_poll=0.5) + return s, exit, pid === StandaloneZODB/ZEO/tests/testZEO.py 1.9 => 1.10 === import os import random +import socket import sys import tempfile import time @@ -163,7 +164,57 @@ # file storage appears to create four files for ext in '', '.index', '.lock', '.tmp': path = self.__fs_base + ext - os.unlink(path) + try: + os.remove(path) + except os.error: + pass + +class WindowsGenericTests(GenericTests): + """Subclass to support server creation on Windows. + + On Windows, the getStorage() design won't work because the storage + can't be created in the parent process and passed to the child. + All the work has to be done in the server's process. + """ + __super_setUp = StorageTestBase.StorageTestBase.setUp + __super_tearDown = StorageTestBase.StorageTestBase.tearDown + + def setUp(self): + self.__super_setUp() + args = self.getStorageInfo() + name = args[0] + args = args[1:] + zeo_addr, self.test_addr, self.test_pid = \ + forker.start_zeo_server(name, args) + storage = ZEO.ClientStorage.ClientStorage(zeo_addr, debug=1, + min_disconnect_poll=0.5) + self._storage = PackWaitWrapper(storage) + storage.registerDB(DummyDB(), None) + + def tearDown(self): + self._storage.close() + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_addr) + # the connection should cause the storage server to die +## os.waitpid(self.test_pid, 0) + time.sleep(0.5) + self.delStorage() + self.__super_tearDown() + +class WindowsZEOFileStorageTests(WindowsGenericTests): + + def getStorageInfo(self): + self.__fs_base = tempfile.mktemp() + return 'FileStorage', self.__fs_base, '1' + + def delStorage(self): + # file storage appears to create four files + for ext in '', '.index', '.lock', '.tmp': + path = self.__fs_base + ext + try: + os.remove(path) + except os.error: + pass class ConnectionTests(ZEOTestBase): """Tests that explicitly manage the server process. @@ -296,6 +347,13 @@ meth[k] = 1 return meth.keys() +if os.name == "posix": + test_classes = ZEOFileStorageTests, ConnectionTests +elif os.name == "nt": + test_classes = WindowsZEOFileStorageTests +else: + raise RuntimeError, "unsupported os: %s" % os.name + def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname @@ -304,6 +362,9 @@ if meth.startswith(name): suite.addTest(klass(meth)) return suite + +def test_suite(): + return unittest.makeSuite(WindowsZEOFileStorageTests, 'check') def main(): import sys, getopt From jeremy at zope.com Fri Aug 17 15:16:55 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - forker.py:1.7 testZEO.py:1.11 winserver.py:1.2 Message-ID: <200108171916.f7HJGt907270@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv6800 Modified Files: forker.py testZEO.py winserver.py Log Message: Update ZEO connection tests to run on Windows forker.py: Extend start_zeo_server() on Windows with arg that specifies port, which is used to restart the server on the same port. Remove unnecessary imports. Set PYTHONPATH for the spawned script. testZEO.py: Refactor ConnectionTests into abstract base class with two platform- specific subclasses. Add goofy test_suite() function, which is goofy because I can't figure out how to use unittest any other way. winserver.py: Keep track of storage in ZEOTestServer so that its close() method can be called when the server exits. === StandaloneZODB/ZEO/tests/forker.py 1.6 => 1.7 === import asyncore -import atexit import os import profile import random -import socket import sys -import threading -import time import types -import ThreadedAsync import ZEO.ClientStorage, ZEO.StorageServer PROFILE = 0 if os.name == "nt": - def start_zeo_server(storage_name, args): + def start_zeo_server(storage_name, args, port=None): """Start a ZEO server in a separate process. Returns the ZEO port, the test server port, and the pid. """ import ZEO.tests.winserver - port = random.randrange(20000, 30000) + if port is None: + port = random.randrange(20000, 30000) script = ZEO.tests.winserver.__file__ if script.endswith('.pyc'): script = script[:-1] args = (sys.executable, script, str(port), storage_name) + args - pid = os.spawnv(os.P_NOWAIT, sys.executable, args) + d = os.environ.copy() + d['PYTHONPATH'] = os.pathsep.join(sys.path) + pid = os.spawnve(os.P_NOWAIT, sys.executable, args, os.environ) return ('localhost', port), ('localhost', port + 1), pid else: === StandaloneZODB/ZEO/tests/testZEO.py 1.10 => 1.11 === from ZODB.tests.StorageTestBase import zodb_unpickle - ZERO = '\0'*8 class DummyDB: @@ -223,44 +222,18 @@ start and stop a ZEO storage server. """ - __super_setUp = StorageTestBase.StorageTestBase.setUp - __super_tearDown = StorageTestBase.StorageTestBase.tearDown - ports = range(29000, 30000, 10) # enough for 100 tests random.shuffle(ports) - def setUp(self): - """Start a ZEO server using a Unix domain socket - - The ZEO server uses the storage object returned by the - getStorage() method. - """ - self.running = 1 - self.__fs_base = tempfile.mktemp() - self.addr = '', random.randrange(2000, 3000) - pid, exit = self._startServer() - self._pid = pid - self._server = exit - self.__super_setUp() - - def _startServer(self, create=1): - fs = FileStorage(self.__fs_base, create=create) - return forker.start_zeo_server(fs, self.addr) + __super_tearDown = StorageTestBase.StorageTestBase.tearDown def openClientStorage(self, cache='', cache_size=200000, wait=1): - base = ZEO.ClientStorage.ClientStorage(self.addr, - client=cache, - cache_size=cache_size, - wait_for_server_on_startup=wait) - storage = PackWaitWrapper(base) - storage.registerDB(DummyDB(), None) - return storage + # defined by subclasses + pass def shutdownServer(self): - if self.running: - self.running = 0 - self._server.close() - os.waitpid(self._pid, 0) + # defined by subclasses + pass def tearDown(self): """Try to cause the tests to halt""" @@ -323,7 +296,7 @@ revid1 = self._dostore(oid, data=obj) self.shutdownServer() self.running = 1 - self._pid, self._server = self._startServer(create=0) + self._startServer(create=0) oid = self._storage.new_oid() obj = MinPO(12) while 1: @@ -335,6 +308,81 @@ else: break +class UnixConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + """Start a ZEO server using a Unix domain socket + + The ZEO server uses the storage object returned by the + getStorage() method. + """ + self.running = 1 + self.__fs_base = tempfile.mktemp() + self.addr = '', self.ports.pop() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + fs = FileStorage(self.__fs_base, create=create) + self._pid, self._server = forker.start_zeo_server(fs, self.addr) + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + self._server.close() + os.waitpid(self._pid, 0) + +class WindowsConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + self.file = tempfile.mktemp() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + if create == 0: + port = self.addr[1] + else: + port = None + self.addr, self.test_a, pid = forker.start_zeo_server('FileStorage', + (self.file, + str(create)), + port) + self.running = 1 + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + debug=1, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_a) + s.close() + time.sleep(1.0) + + def tearDown(self): + self.shutdownServer() + + def get_methods(klass): l = [klass] meth = {} @@ -348,9 +396,9 @@ return meth.keys() if os.name == "posix": - test_classes = ZEOFileStorageTests, ConnectionTests + test_classes = ZEOFileStorageTests, UnixConnectionTests elif os.name == "nt": - test_classes = WindowsZEOFileStorageTests + test_classes = WindowsZEOFileStorageTests, WindowsConnectionTests else: raise RuntimeError, "unsupported os: %s" % os.name @@ -364,7 +412,13 @@ return suite def test_suite(): - return unittest.makeSuite(WindowsZEOFileStorageTests, 'check') + t = unittest.TestLoader() + t.testMethodPrefix = 'check' + tests = [] + for klass in test_classes: + s = t.loadTestsFromTestCase(klass) + tests.extend(s._tests) + return unittest.TestSuite(tests) def main(): import sys, getopt @@ -385,4 +439,5 @@ runner.run(tests) if __name__ == "__main__": + print sys.path main() === StandaloneZODB/ZEO/tests/winserver.py 1.1 => 1.2 === __super_init = asyncore.dispatcher.__init__ - def __init__(self, addr): + def __init__(self, addr, storage): self.__super_init() + self.storage = storage if type(addr) == types.StringType: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) else: @@ -28,6 +29,7 @@ def handle_accept(self): sock, addr = self.accept() + self.storage.close() os._exit(0) def load_storage_class(name): @@ -40,12 +42,11 @@ storage = klass(*args) zeo_port = int(port) test_port = zeo_port + 1 - t = ZEOTestServer(('', test_port)) -## t = threading.Thread(target=ZEOTestServer, args=(('', test_port),)) -## t.start() + t = ZEOTestServer(('', test_port), storage) serv = ZEO.StorageServer.StorageServer(('', zeo_port), {'1': storage}) asyncore.loop() if __name__ == "__main__": import sys + main(sys.argv[1], sys.argv[2], sys.argv[3:]) From jeremy at zope.com Wed Aug 22 17:04:42 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - asyncwrap.py:1.1.2.1 StorageServer.py:1.28.2.3 zrpc.py:1.18.2.2 Message-ID: <200108222104.f7ML4gQ19375@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv19352 Modified Files: Tag: zeo-1_0-branch StorageServer.py zrpc.py Added Files: Tag: zeo-1_0-branch asyncwrap.py Log Message: Use asyncwrap to call asyncore.loop() and asyncore.poll() === Added File StandaloneZODB/ZEO/asyncwrap.py === """A wrapper for asyncore that provides robust exception handling. The poll() and loop() calls exported by asyncore can raise exceptions. asyncore uses either the select() or poll() system call. It is possible for those system calls to fail, returning, for example, EINTR. Python raises a select.error when an error occurs. If the program using asyncore doesn't catch the exception, it will die with an uncaught exception. This module exports safer versions of loop() and poll() that wrap the asyncore calls in try/except handlers that catch the errors and do the right thing. In most cases, it is safe to catch the error and simply retry the call. XXX Operations on asyncore sockets can also fail with exceptions that can safely be caught and ignored by user programs. It's not clear if it would be useful to extend this module with wrappers for those errors. """ # XXX The current implementation requires Python 2.0. Not sure if # that's acceptable, depends on how many users want to combine ZEO 1.0 # and Zope 2.3. import asyncore import errno import select def loop(*args, **kwargs): while 1: try: apply(asyncore.loop, args, kwargs) except select.error, err: if err[0] != errno.EINTR: raise else: break def poll(*args, **kwargs): try: apply(asyncore.poll, args, kwargs) except select.error, err: if err[0] != errno.EINTR: raise === StandaloneZODB/ZEO/StorageServer.py 1.28.2.2 => 1.28.2.3 === from cStringIO import StringIO from ZEO import trigger +from ZEO import asyncwrap class StorageServerError(POSException.StorageError): pass @@ -184,8 +185,8 @@ sock, addr = self.accept() except socket.error: sys.stderr.write('warning: accept failed\n') - - ZEOConnection(self, sock, addr) + else: + ZEOConnection(self, sock, addr) def log_info(self, message, type='info'): if type=='error': type=ERROR @@ -577,7 +578,10 @@ import ZODB.FileStorage name, port = sys.argv[1:3] blather(name, port) - try: port='',string.atoi(port) - except: pass + try: + port='', int(port) + except: + pass + StorageServer(port, ZODB.FileStorage.FileStorage(name)) - asyncore.loop() + asyncwrap.loop() === StandaloneZODB/ZEO/zrpc.py 1.18.2.1 => 1.18.2.2 === from smac import SizedMessageAsyncConnection import socket, string, struct, asyncore, sys, time, select -TupleType=type(()) from zLOG import LOG, TRACE, DEBUG, INFO +from ZEO import asyncwrap from errno import EINTR +TupleType=type(()) # We create a special fast pickler! This allows us # to create slightly more efficient pickles and @@ -186,13 +187,13 @@ try: r, w, e = select.select([self._fileno],[],[],0.0) except select.error, v: if v[0] != EINTR: raise - if r: asyncore.poll(0.0, self) + if r: asyncwrap.poll(0.0, self) else: break def readLoop(self): la=self.__la while not la(0): - asyncore.poll(60.0, self) + asyncwrap.poll(60.0, self) self.__lr() def setLoop(self, map=None, Wakeup=lambda : None): @@ -242,7 +243,7 @@ self.message_output(dump(args,1)) if self.__haveMainLoop: self.__Wakeup() # Wake up the main loop - else: asyncore.poll(0.0, self) + else: asyncwrap.poll(0.0, self) def setOutOfBand(self, f): """Define a call-back function for handling out-of-band communication From jeremy at zope.com Wed Aug 22 17:05:31 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - smac.py:1.10.2.1 Message-ID: <200108222105.f7ML5VA19449@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv19433 Modified Files: Tag: zeo-1_0-branch smac.py Log Message: Wrap send() and recv() calls in try/except The try/except catches errors like EAGAIN that indicate transient failures. In those cases, treat as send() and recv() of no data and return from method gracefully. === StandaloneZODB/ZEO/smac.py 1.10 => 1.10.2.1 === import asyncore, string, struct, zLOG, sys, Acquisition +import socket, errno from zLOG import LOG, TRACE, ERROR, INFO +# Use the dictionary to make sure we get the minimum number of errno +# entries. We expect that EWOULDBLOCK == EAGAIN on most systems -- +# or that only one is actually used. + +tmp_dict = {errno.EWOULDBLOCK: 0, + errno.EAGAIN: 0, + errno.EINTR: 0, + } +expected_socket_read_errors = tuple(tmp_dict.keys()) + +tmp_dict = {errno.EAGAIN: 0, + errno.EWOULDBLOCK: 0, + errno.ENOBUFS: 0, + errno.EINTR: 0, + } +expected_socket_write_errors = tuple(tmp_dict.keys()) +del tmp_dict + class SizedMessageAsyncConnection(Acquisition.Explicit, asyncore.dispatcher): __append=None # Marker indicating that we're closed @@ -116,7 +135,12 @@ join=string.join, StringType=type(''), _type=type, _None=None): - d=self.recv(8096) + try: + d=self.recv(8096) + except socket.error, err: + if err[0] in expected_socket_read_errors: + return + raise if not d: return inp=self.__inp @@ -160,7 +184,12 @@ output=self.__output while output: v=output[0] - n=self.send(v) + try: + n=self.send(v) + except socket.error, err: + if err[0] in expected_socket_write_errors: + break # we couldn't write anything + raise if n < len(v): output[0]=v[n:] break # we can't write any more From jeremy at zope.com Thu Aug 23 13:47:08 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO - CHANGES.txt:1.25.2.4 Message-ID: <200108231747.f7NHl8N06128@cvs.baymountain.com> Update of /cvs-repository/ZEO In directory cvs.zope.org:/tmp/cvs-serv6117 Modified Files: Tag: zeo-1_0-branch CHANGES.txt Log Message: Update changelog for 1.0b4 === ZEO/CHANGES.txt 1.25.2.3 => 1.25.2.4 === Bugs fixed + - Improved robustness of ZEO under load. Many non-fatal socket + errors are caught and handled now. Also fix NameError bug that + could cause ZEO to fail on Windows. + + - Reduced the likelihood that a thread.error exception ("release + unlocked lock") will be raised if a client disconnects from a + storage unexpectedly. + - Fixed a bug in undo handling that caused connections to be reset and prevented undos from propagating to all clients. From jeremy at zope.com Thu Aug 23 14:31:14 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.12 Message-ID: <200108231831.f7NIVEj08190@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv8169 Modified Files: testZEO.py Log Message: remove debugging print === ZEO/ZEO/tests/testZEO.py 1.11 => 1.12 === if __name__ == "__main__": - print sys.path main() From jeremy at zope.com Thu Aug 23 14:31:14 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.12 Message-ID: <200108231831.f7NIVE308195@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv8169 Modified Files: testZEO.py Log Message: remove debugging print === ZEO/ZEO/tests/testZEO.py 1.11 => 1.12 === if __name__ == "__main__": - print sys.path main() From jeremy at zope.com Thu Aug 23 15:29:49 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - forker.py:1.5.2.2 multi.py:1.3.2.1 speed.py:1.4.2.1 testZEO.py:1.9.2.2 Message-ID: <200108231929.f7NJTnB11010@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv10976 Modified Files: Tag: zeo-1_0-branch forker.py multi.py speed.py testZEO.py Log Message: Remove Python 2.x-isms so that tests run under Python 1.5.2 === ZEO/ZEO/tests/forker.py 1.5.2.1 => 1.5.2.2 === import asyncore -import atexit import os import profile import sys === ZEO/ZEO/tests/multi.py 1.3 => 1.3.2.1 === server_pid, server = start_server(addr) t1 = time.time() - pids = [start_client(addr, client_func) for i in range(CLIENTS)] + pids = [] + for i in range(CLIENTS): + pids.append(start_client(addr, client_func)) for pid in pids: assert type(pid) == types.IntType, "invalid pid type: %s (%s)" % \ (repr(pid), type(pid)) === ZEO/ZEO/tests/speed.py 1.4 => 1.4.2.1 === if threads > 1: import threading - l = [threading.Thread(target=work, - args=(db, results, nrep, compress, data, - detailed, minimize, i)) - for i in range(threads)] + l = [] + for i in range(threads): + t = threading.Thread(target=work, + args=(db, results, nrep, compress, data, + detailed, minimize, i)) + l.append(t) for t in l: t.start() for t in l: @@ -263,7 +265,9 @@ print '-'*24 print "num\tmean\tmin\tmax" for r in 1, 10, 100, 1000: - times = [time for time, conf in results[r]] + times = [] + for time, conf in results[r]: + times.append(time) t = mean(times) print "%d\t%.4f\t%.4f\t%.4f" % (r, t, min(times), max(times)) === ZEO/ZEO/tests/testZEO.py 1.9.2.1 => 1.9.2.2 === __super_tearDown = StorageTestBase.StorageTestBase.tearDown - ports = [random.randrange(25000, 30000) for i in range(200)] + ports = [] + for i in range(200): + ports.append(random.randrange(25000, 30000)) + del i def setUp(self): """Start a ZEO server using a Unix domain socket @@ -186,7 +189,8 @@ """ self.running = 1 self.__fs_base = tempfile.mktemp() - port = self.ports.pop() + port = self.ports[0] + del self.ports[0] self.addr = '', port pid, exit = self._startServer() self._pid = pid @@ -306,9 +310,10 @@ def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname + lname = len(name) for klass in ZEOFileStorageTests, ConnectionTests: for meth in get_methods(klass): - if meth.startswith(name): + if meth[:lname] == name: suite.addTest(klass(meth)) return suite From jeremy at zope.com Thu Aug 23 15:29:49 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - forker.py:1.5.2.2 multi.py:1.3.2.1 speed.py:1.4.2.1 testZEO.py:1.9.2.2 Message-ID: <200108231929.f7NJTn911017@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv10976 Modified Files: Tag: zeo-1_0-branch forker.py multi.py speed.py testZEO.py Log Message: Remove Python 2.x-isms so that tests run under Python 1.5.2 === ZEO/ZEO/tests/forker.py 1.5.2.1 => 1.5.2.2 === import asyncore -import atexit import os import profile import sys === ZEO/ZEO/tests/multi.py 1.3 => 1.3.2.1 === server_pid, server = start_server(addr) t1 = time.time() - pids = [start_client(addr, client_func) for i in range(CLIENTS)] + pids = [] + for i in range(CLIENTS): + pids.append(start_client(addr, client_func)) for pid in pids: assert type(pid) == types.IntType, "invalid pid type: %s (%s)" % \ (repr(pid), type(pid)) === ZEO/ZEO/tests/speed.py 1.4 => 1.4.2.1 === if threads > 1: import threading - l = [threading.Thread(target=work, - args=(db, results, nrep, compress, data, - detailed, minimize, i)) - for i in range(threads)] + l = [] + for i in range(threads): + t = threading.Thread(target=work, + args=(db, results, nrep, compress, data, + detailed, minimize, i)) + l.append(t) for t in l: t.start() for t in l: @@ -263,7 +265,9 @@ print '-'*24 print "num\tmean\tmin\tmax" for r in 1, 10, 100, 1000: - times = [time for time, conf in results[r]] + times = [] + for time, conf in results[r]: + times.append(time) t = mean(times) print "%d\t%.4f\t%.4f\t%.4f" % (r, t, min(times), max(times)) === ZEO/ZEO/tests/testZEO.py 1.9.2.1 => 1.9.2.2 === __super_tearDown = StorageTestBase.StorageTestBase.tearDown - ports = [random.randrange(25000, 30000) for i in range(200)] + ports = [] + for i in range(200): + ports.append(random.randrange(25000, 30000)) + del i def setUp(self): """Start a ZEO server using a Unix domain socket @@ -186,7 +189,8 @@ """ self.running = 1 self.__fs_base = tempfile.mktemp() - port = self.ports.pop() + port = self.ports[0] + del self.ports[0] self.addr = '', port pid, exit = self._startServer() self._pid = pid @@ -306,9 +310,10 @@ def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname + lname = len(name) for klass in ZEOFileStorageTests, ConnectionTests: for meth in get_methods(klass): - if meth.startswith(name): + if meth[:lname] == name: suite.addTest(klass(meth)) return suite From jeremy at zope.com Thu Aug 23 20:56:18 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.13 Message-ID: <200108240056.f7O0uIY25347@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25326 Modified Files: testZEO.py Log Message: Merge updates for Windows test success from the trunk === ZEO/ZEO/tests/testZEO.py 1.12 => 1.13 === """ - ports = range(29000, 30000, 10) # enough for 100 tests - random.shuffle(ports) + __super_setUp = StorageTestBase.StorageTestBase.setUp - __super_tearDown = StorageTestBase.StorageTestBase.tearDown + ports = [] + for i in range(200): + ports.append(random.randrange(25000, 30000)) + del i def openClientStorage(self, cache='', cache_size=200000, wait=1): # defined by subclasses @@ -302,11 +304,16 @@ while 1: try: revid1 = self._dostore(oid, data=obj) - except (ClientDisconnected, thread.error), err: + except (ClientDisconnected, thread.error, socket.error), err: get_transaction().abort() time.sleep(0.1) else: break + # XXX This is a bloody pain. We're placing a heavy burden + # on users to catch a plethora of exceptions in order to + # write robust code. Need to think about implementing + # John Heintz's suggestion to make sure all exceptions + # inherit from POSException. class UnixConnectionTests(ConnectionTests): __super_setUp = StorageTestBase.StorageTestBase.setUp @@ -381,8 +388,8 @@ def tearDown(self): self.shutdownServer() - - + + def get_methods(klass): l = [klass] meth = {} @@ -405,20 +412,15 @@ def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname - for klass in ZEOFileStorageTests, ConnectionTests: + lname = len(name) + for klass in test_classes: for meth in get_methods(klass): - if meth.startswith(name): + if meth[:lname] == name: suite.addTest(klass(meth)) return suite def test_suite(): - t = unittest.TestLoader() - t.testMethodPrefix = 'check' - tests = [] - for klass in test_classes: - s = t.loadTestsFromTestCase(klass) - tests.extend(s._tests) - return unittest.TestSuite(tests) + return makeTestSuite() def main(): import sys, getopt From jeremy at zope.com Thu Aug 23 20:56:18 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.13 Message-ID: <200108240056.f7O0uIR25352@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25326 Modified Files: testZEO.py Log Message: Merge updates for Windows test success from the trunk === ZEO/ZEO/tests/testZEO.py 1.12 => 1.13 === """ - ports = range(29000, 30000, 10) # enough for 100 tests - random.shuffle(ports) + __super_setUp = StorageTestBase.StorageTestBase.setUp - __super_tearDown = StorageTestBase.StorageTestBase.tearDown + ports = [] + for i in range(200): + ports.append(random.randrange(25000, 30000)) + del i def openClientStorage(self, cache='', cache_size=200000, wait=1): # defined by subclasses @@ -302,11 +304,16 @@ while 1: try: revid1 = self._dostore(oid, data=obj) - except (ClientDisconnected, thread.error), err: + except (ClientDisconnected, thread.error, socket.error), err: get_transaction().abort() time.sleep(0.1) else: break + # XXX This is a bloody pain. We're placing a heavy burden + # on users to catch a plethora of exceptions in order to + # write robust code. Need to think about implementing + # John Heintz's suggestion to make sure all exceptions + # inherit from POSException. class UnixConnectionTests(ConnectionTests): __super_setUp = StorageTestBase.StorageTestBase.setUp @@ -381,8 +388,8 @@ def tearDown(self): self.shutdownServer() - - + + def get_methods(klass): l = [klass] meth = {} @@ -405,20 +412,15 @@ def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname - for klass in ZEOFileStorageTests, ConnectionTests: + lname = len(name) + for klass in test_classes: for meth in get_methods(klass): - if meth.startswith(name): + if meth[:lname] == name: suite.addTest(klass(meth)) return suite def test_suite(): - t = unittest.TestLoader() - t.testMethodPrefix = 'check' - tests = [] - for klass in test_classes: - s = t.loadTestsFromTestCase(klass) - tests.extend(s._tests) - return unittest.TestSuite(tests) + return makeTestSuite() def main(): import sys, getopt From jeremy at zope.com Thu Aug 23 20:58:07 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - winserver.py:1.2.2.1 Message-ID: <200108240058.f7O0w7h25500@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25479 Added Files: Tag: zeo-1_0-branch winserver.py Log Message: Adding file from trunk (for Windows test success) === Added File ZEO/ZEO/tests/winserver.py === """Helper file used to launch ZEO server for Windows tests""" import asyncore import os import random import socket import threading import types import ZEO.StorageServer class ZEOTestServer(asyncore.dispatcher): """A trivial server for killing a server at the end of a test The server calls os._exit() as soon as it is connected to. No chance to even send some data down the socket. """ __super_init = asyncore.dispatcher.__init__ def __init__(self, addr, storage): self.__super_init() self.storage = storage if type(addr) == types.StringType: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) else: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.bind(addr) self.listen(5) def handle_accept(self): sock, addr = self.accept() self.storage.close() os._exit(0) def load_storage_class(name): package = __import__("ZODB." + name) mod = getattr(package, name) return getattr(mod, name) def main(port, storage_name, args): klass = load_storage_class(storage_name) storage = klass(*args) zeo_port = int(port) test_port = zeo_port + 1 t = ZEOTestServer(('', test_port), storage) serv = ZEO.StorageServer.StorageServer(('', zeo_port), {'1': storage}) asyncore.loop() if __name__ == "__main__": import sys main(sys.argv[1], sys.argv[2], sys.argv[3:]) From jeremy at zope.com Thu Aug 23 20:58:07 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - winserver.py:1.2.2.1 Message-ID: <200108240058.f7O0w7B25505@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25479 Added Files: Tag: zeo-1_0-branch winserver.py Log Message: Adding file from trunk (for Windows test success) === Added File ZEO/ZEO/tests/winserver.py === """Helper file used to launch ZEO server for Windows tests""" import asyncore import os import random import socket import threading import types import ZEO.StorageServer class ZEOTestServer(asyncore.dispatcher): """A trivial server for killing a server at the end of a test The server calls os._exit() as soon as it is connected to. No chance to even send some data down the socket. """ __super_init = asyncore.dispatcher.__init__ def __init__(self, addr, storage): self.__super_init() self.storage = storage if type(addr) == types.StringType: self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM) else: self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.bind(addr) self.listen(5) def handle_accept(self): sock, addr = self.accept() self.storage.close() os._exit(0) def load_storage_class(name): package = __import__("ZODB." + name) mod = getattr(package, name) return getattr(mod, name) def main(port, storage_name, args): klass = load_storage_class(storage_name) storage = klass(*args) zeo_port = int(port) test_port = zeo_port + 1 t = ZEOTestServer(('', test_port), storage) serv = ZEO.StorageServer.StorageServer(('', zeo_port), {'1': storage}) asyncore.loop() if __name__ == "__main__": import sys main(sys.argv[1], sys.argv[2], sys.argv[3:]) From jeremy at zope.com Thu Aug 23 20:59:04 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - forker.py:1.5.2.3 testZEO.py:1.9.2.3 Message-ID: <200108240059.f7O0x4n25586@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25555 Modified Files: Tag: zeo-1_0-branch forker.py testZEO.py Log Message: Merge changes from trunk (for Windows test success) === ZEO/ZEO/tests/forker.py 1.5.2.2 => 1.5.2.3 === import os import profile +import random import sys -import time import types -import ThreadedAsync import ZEO.ClientStorage, ZEO.StorageServer PROFILE = 0 -class ZEOServerExit(asyncore.file_dispatcher): - """Used to exit ZEO.StorageServer when run is done""" +if os.name == "nt": - def writable(self): - return 0 + def start_zeo_server(storage_name, args, port=None): + """Start a ZEO server in a separate process. - def readable(self): - return 1 + Returns the ZEO port, the test server port, and the pid. + """ + import ZEO.tests.winserver + if port is None: + port = random.randrange(20000, 30000) + script = ZEO.tests.winserver.__file__ + if script.endswith('.pyc'): + script = script[:-1] + args = (sys.executable, script, str(port), storage_name) + args + d = os.environ.copy() + d['PYTHONPATH'] = os.pathsep.join(sys.path) + pid = os.spawnve(os.P_NOWAIT, sys.executable, args, os.environ) + return ('localhost', port), ('localhost', port + 1), pid + +else: + + class ZEOServerExit(asyncore.file_dispatcher): + """Used to exit ZEO.StorageServer when run is done""" + + def writable(self): + return 0 + + def readable(self): + return 1 + + def handle_read(self): + buf = self.recv(4) + if buf: + assert buf == "done" + asyncore.socket_map.clear() - def handle_read(self): - buf = self.recv(4) - if buf: - assert buf == "done" + def handle_close(self): asyncore.socket_map.clear() - - def handle_close(self): - asyncore.socket_map.clear() - -class ZEOClientExit: - """Used by client to cause server to exit""" - def __init__(self, pipe): - self.pipe = pipe - - def close(self): - os.write(self.pipe, "done") - -def start_zeo_server(storage, addr): - rd, wr = os.pipe() - pid = os.fork() - if pid == 0: - if PROFILE: - p = profile.Profile() - p.runctx("run_server(storage, addr, rd, wr)", globals(), - locals()) - p.dump_stats("stats.s.%d" % os.getpid()) - else: - run_server(storage, addr, rd, wr) - os._exit(0) - else: - os.close(rd) - return pid, ZEOClientExit(wr) - -def run_server(storage, addr, rd, wr): - # in the child, run the storage server - os.close(wr) - ZEOServerExit(rd) - serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) - asyncore.loop() - storage.close() - if isinstance(addr, types.StringType): - os.unlink(addr) - -def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", - storage_id="1", cache_size=20000000): - """Setup ZEO client-server for storage. - - Returns a ClientStorage instance and a ZEOClientExit instance. - - XXX Don't know if os.pipe() will work on Windows. - """ - - if domain == "AF_INET": - import random - addr = '', random.randrange(25000, 30000) - elif domain == "AF_UNIX": - import tempfile - addr = tempfile.mktemp() - else: - raise ValueError, "bad domain: %s" % domain - - for i in 1, 2, 3: - try: - pid, exit = start_zeo_server(storage, addr) - except socket.error, (num, msg): - if num == 98: - continue + + class ZEOClientExit: + """Used by client to cause server to exit""" + def __init__(self, pipe): + self.pipe = pipe + + def close(self): + os.write(self.pipe, "done") + + def start_zeo_server(storage, addr): + rd, wr = os.pipe() + pid = os.fork() + if pid == 0: + if PROFILE: + p = profile.Profile() + p.runctx("run_server(storage, addr, rd, wr)", globals(), + locals()) + p.dump_stats("stats.s.%d" % os.getpid()) else: - raise + run_server(storage, addr, rd, wr) + os._exit(0) else: - break - s = ZEO.ClientStorage.ClientStorage(addr, storage_id, - debug=1, client=cache, - cache_size=cache_size, - min_disconnect_poll=0.5) - return s, exit, pid + os.close(rd) + return pid, ZEOClientExit(wr) + + def run_server(storage, addr, rd, wr): + # in the child, run the storage server + os.close(wr) + ZEOServerExit(rd) + serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) + asyncore.loop() + storage.close() + if isinstance(addr, types.StringType): + os.unlink(addr) + + def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", + storage_id="1", cache_size=20000000): + """Setup ZEO client-server for storage. + + Returns a ClientStorage instance and a ZEOClientExit instance. + + XXX Don't know if os.pipe() will work on Windows. + """ + + if domain == "AF_INET": + import random + addr = '', random.randrange(2000, 3000) + elif domain == "AF_UNIX": + import tempfile + addr = tempfile.mktemp() + else: + raise ValueError, "bad domain: %s" % domain + + pid, exit = start_zeo_server(storage, addr) + s = ZEO.ClientStorage.ClientStorage(addr, storage_id, + debug=1, client=cache, + cache_size=cache_size, + min_disconnect_poll=0.5) + return s, exit, pid === ZEO/ZEO/tests/testZEO.py 1.9.2.2 => 1.9.2.3 === from ZODB.tests.StorageTestBase import zodb_unpickle - ZERO = '\0'*8 class DummyDB: @@ -164,7 +163,57 @@ # file storage appears to create four files for ext in '', '.index', '.lock', '.tmp': path = self.__fs_base + ext - os.unlink(path) + try: + os.remove(path) + except os.error: + pass + +class WindowsGenericTests(GenericTests): + """Subclass to support server creation on Windows. + + On Windows, the getStorage() design won't work because the storage + can't be created in the parent process and passed to the child. + All the work has to be done in the server's process. + """ + __super_setUp = StorageTestBase.StorageTestBase.setUp + __super_tearDown = StorageTestBase.StorageTestBase.tearDown + + def setUp(self): + self.__super_setUp() + args = self.getStorageInfo() + name = args[0] + args = args[1:] + zeo_addr, self.test_addr, self.test_pid = \ + forker.start_zeo_server(name, args) + storage = ZEO.ClientStorage.ClientStorage(zeo_addr, debug=1, + min_disconnect_poll=0.5) + self._storage = PackWaitWrapper(storage) + storage.registerDB(DummyDB(), None) + + def tearDown(self): + self._storage.close() + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_addr) + # the connection should cause the storage server to die +## os.waitpid(self.test_pid, 0) + time.sleep(0.5) + self.delStorage() + self.__super_tearDown() + +class WindowsZEOFileStorageTests(WindowsGenericTests): + + def getStorageInfo(self): + self.__fs_base = tempfile.mktemp() + return 'FileStorage', self.__fs_base, '1' + + def delStorage(self): + # file storage appears to create four files + for ext in '', '.index', '.lock', '.tmp': + path = self.__fs_base + ext + try: + os.remove(path) + except os.error: + pass class ConnectionTests(ZEOTestBase): """Tests that explicitly manage the server process. @@ -174,47 +223,19 @@ """ __super_setUp = StorageTestBase.StorageTestBase.setUp - __super_tearDown = StorageTestBase.StorageTestBase.tearDown ports = [] for i in range(200): ports.append(random.randrange(25000, 30000)) del i - def setUp(self): - """Start a ZEO server using a Unix domain socket - - The ZEO server uses the storage object returned by the - getStorage() method. - """ - self.running = 1 - self.__fs_base = tempfile.mktemp() - port = self.ports[0] - del self.ports[0] - self.addr = '', port - pid, exit = self._startServer() - self._pid = pid - self._server = exit - self.__super_setUp() - - def _startServer(self, create=1): - fs = FileStorage(self.__fs_base, create=create) - return forker.start_zeo_server(fs, self.addr) - def openClientStorage(self, cache='', cache_size=200000, wait=1): - base = ZEO.ClientStorage.ClientStorage(self.addr, - client=cache, - cache_size=cache_size, - wait_for_server_on_startup=wait) - storage = PackWaitWrapper(base) - storage.registerDB(DummyDB(), None) - return storage + # defined by subclasses + pass def shutdownServer(self): - if self.running: - self.running = 0 - self._server.close() - os.waitpid(self._pid, 0) + # defined by subclasses + pass def tearDown(self): """Try to cause the tests to halt""" @@ -277,15 +298,9 @@ revid1 = self._dostore(oid, data=obj) self.shutdownServer() self.running = 1 - self._pid, self._server = self._startServer(create=0) + self._startServer(create=0) oid = self._storage.new_oid() obj = MinPO(12) - # XXX This is a bloody pain. We're placing a heavy burden on - # users to catch a plethora of exceptions in order to write - # robust code. Need to think about implementing John Heintz's - # suggestion to make sure all exceptions inherit from - # POSException. - while 1: try: revid1 = self._dostore(oid, data=obj) @@ -294,6 +309,86 @@ time.sleep(0.1) else: break + # XXX This is a bloody pain. We're placing a heavy burden + # on users to catch a plethora of exceptions in order to + # write robust code. Need to think about implementing + # John Heintz's suggestion to make sure all exceptions + # inherit from POSException. + +class UnixConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + """Start a ZEO server using a Unix domain socket + + The ZEO server uses the storage object returned by the + getStorage() method. + """ + self.running = 1 + self.__fs_base = tempfile.mktemp() + self.addr = '', self.ports.pop() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + fs = FileStorage(self.__fs_base, create=create) + self._pid, self._server = forker.start_zeo_server(fs, self.addr) + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + self._server.close() + os.waitpid(self._pid, 0) + +class WindowsConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + self.file = tempfile.mktemp() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + if create == 0: + port = self.addr[1] + else: + port = None + self.addr, self.test_a, pid = forker.start_zeo_server('FileStorage', + (self.file, + str(create)), + port) + self.running = 1 + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + debug=1, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_a) + s.close() + time.sleep(1.0) + + def tearDown(self): + self.shutdownServer() + def get_methods(klass): l = [klass] @@ -307,11 +402,18 @@ meth[k] = 1 return meth.keys() +if os.name == "posix": + test_classes = ZEOFileStorageTests, UnixConnectionTests +elif os.name == "nt": + test_classes = WindowsZEOFileStorageTests, WindowsConnectionTests +else: + raise RuntimeError, "unsupported os: %s" % os.name + def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname lname = len(name) - for klass in ZEOFileStorageTests, ConnectionTests: + for klass in test_classes: for meth in get_methods(klass): if meth[:lname] == name: suite.addTest(klass(meth)) From jeremy at zope.com Thu Aug 23 20:59:05 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - forker.py:1.5.2.3 testZEO.py:1.9.2.3 Message-ID: <200108240059.f7O0x5J25591@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv25555 Modified Files: Tag: zeo-1_0-branch forker.py testZEO.py Log Message: Merge changes from trunk (for Windows test success) === ZEO/ZEO/tests/forker.py 1.5.2.2 => 1.5.2.3 === import os import profile +import random import sys -import time import types -import ThreadedAsync import ZEO.ClientStorage, ZEO.StorageServer PROFILE = 0 -class ZEOServerExit(asyncore.file_dispatcher): - """Used to exit ZEO.StorageServer when run is done""" +if os.name == "nt": - def writable(self): - return 0 + def start_zeo_server(storage_name, args, port=None): + """Start a ZEO server in a separate process. - def readable(self): - return 1 + Returns the ZEO port, the test server port, and the pid. + """ + import ZEO.tests.winserver + if port is None: + port = random.randrange(20000, 30000) + script = ZEO.tests.winserver.__file__ + if script.endswith('.pyc'): + script = script[:-1] + args = (sys.executable, script, str(port), storage_name) + args + d = os.environ.copy() + d['PYTHONPATH'] = os.pathsep.join(sys.path) + pid = os.spawnve(os.P_NOWAIT, sys.executable, args, os.environ) + return ('localhost', port), ('localhost', port + 1), pid + +else: + + class ZEOServerExit(asyncore.file_dispatcher): + """Used to exit ZEO.StorageServer when run is done""" + + def writable(self): + return 0 + + def readable(self): + return 1 + + def handle_read(self): + buf = self.recv(4) + if buf: + assert buf == "done" + asyncore.socket_map.clear() - def handle_read(self): - buf = self.recv(4) - if buf: - assert buf == "done" + def handle_close(self): asyncore.socket_map.clear() - - def handle_close(self): - asyncore.socket_map.clear() - -class ZEOClientExit: - """Used by client to cause server to exit""" - def __init__(self, pipe): - self.pipe = pipe - - def close(self): - os.write(self.pipe, "done") - -def start_zeo_server(storage, addr): - rd, wr = os.pipe() - pid = os.fork() - if pid == 0: - if PROFILE: - p = profile.Profile() - p.runctx("run_server(storage, addr, rd, wr)", globals(), - locals()) - p.dump_stats("stats.s.%d" % os.getpid()) - else: - run_server(storage, addr, rd, wr) - os._exit(0) - else: - os.close(rd) - return pid, ZEOClientExit(wr) - -def run_server(storage, addr, rd, wr): - # in the child, run the storage server - os.close(wr) - ZEOServerExit(rd) - serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) - asyncore.loop() - storage.close() - if isinstance(addr, types.StringType): - os.unlink(addr) - -def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", - storage_id="1", cache_size=20000000): - """Setup ZEO client-server for storage. - - Returns a ClientStorage instance and a ZEOClientExit instance. - - XXX Don't know if os.pipe() will work on Windows. - """ - - if domain == "AF_INET": - import random - addr = '', random.randrange(25000, 30000) - elif domain == "AF_UNIX": - import tempfile - addr = tempfile.mktemp() - else: - raise ValueError, "bad domain: %s" % domain - - for i in 1, 2, 3: - try: - pid, exit = start_zeo_server(storage, addr) - except socket.error, (num, msg): - if num == 98: - continue + + class ZEOClientExit: + """Used by client to cause server to exit""" + def __init__(self, pipe): + self.pipe = pipe + + def close(self): + os.write(self.pipe, "done") + + def start_zeo_server(storage, addr): + rd, wr = os.pipe() + pid = os.fork() + if pid == 0: + if PROFILE: + p = profile.Profile() + p.runctx("run_server(storage, addr, rd, wr)", globals(), + locals()) + p.dump_stats("stats.s.%d" % os.getpid()) else: - raise + run_server(storage, addr, rd, wr) + os._exit(0) else: - break - s = ZEO.ClientStorage.ClientStorage(addr, storage_id, - debug=1, client=cache, - cache_size=cache_size, - min_disconnect_poll=0.5) - return s, exit, pid + os.close(rd) + return pid, ZEOClientExit(wr) + + def run_server(storage, addr, rd, wr): + # in the child, run the storage server + os.close(wr) + ZEOServerExit(rd) + serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) + asyncore.loop() + storage.close() + if isinstance(addr, types.StringType): + os.unlink(addr) + + def start_zeo(storage, cache=None, cleanup=None, domain="AF_INET", + storage_id="1", cache_size=20000000): + """Setup ZEO client-server for storage. + + Returns a ClientStorage instance and a ZEOClientExit instance. + + XXX Don't know if os.pipe() will work on Windows. + """ + + if domain == "AF_INET": + import random + addr = '', random.randrange(2000, 3000) + elif domain == "AF_UNIX": + import tempfile + addr = tempfile.mktemp() + else: + raise ValueError, "bad domain: %s" % domain + + pid, exit = start_zeo_server(storage, addr) + s = ZEO.ClientStorage.ClientStorage(addr, storage_id, + debug=1, client=cache, + cache_size=cache_size, + min_disconnect_poll=0.5) + return s, exit, pid === ZEO/ZEO/tests/testZEO.py 1.9.2.2 => 1.9.2.3 === from ZODB.tests.StorageTestBase import zodb_unpickle - ZERO = '\0'*8 class DummyDB: @@ -164,7 +163,57 @@ # file storage appears to create four files for ext in '', '.index', '.lock', '.tmp': path = self.__fs_base + ext - os.unlink(path) + try: + os.remove(path) + except os.error: + pass + +class WindowsGenericTests(GenericTests): + """Subclass to support server creation on Windows. + + On Windows, the getStorage() design won't work because the storage + can't be created in the parent process and passed to the child. + All the work has to be done in the server's process. + """ + __super_setUp = StorageTestBase.StorageTestBase.setUp + __super_tearDown = StorageTestBase.StorageTestBase.tearDown + + def setUp(self): + self.__super_setUp() + args = self.getStorageInfo() + name = args[0] + args = args[1:] + zeo_addr, self.test_addr, self.test_pid = \ + forker.start_zeo_server(name, args) + storage = ZEO.ClientStorage.ClientStorage(zeo_addr, debug=1, + min_disconnect_poll=0.5) + self._storage = PackWaitWrapper(storage) + storage.registerDB(DummyDB(), None) + + def tearDown(self): + self._storage.close() + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_addr) + # the connection should cause the storage server to die +## os.waitpid(self.test_pid, 0) + time.sleep(0.5) + self.delStorage() + self.__super_tearDown() + +class WindowsZEOFileStorageTests(WindowsGenericTests): + + def getStorageInfo(self): + self.__fs_base = tempfile.mktemp() + return 'FileStorage', self.__fs_base, '1' + + def delStorage(self): + # file storage appears to create four files + for ext in '', '.index', '.lock', '.tmp': + path = self.__fs_base + ext + try: + os.remove(path) + except os.error: + pass class ConnectionTests(ZEOTestBase): """Tests that explicitly manage the server process. @@ -174,47 +223,19 @@ """ __super_setUp = StorageTestBase.StorageTestBase.setUp - __super_tearDown = StorageTestBase.StorageTestBase.tearDown ports = [] for i in range(200): ports.append(random.randrange(25000, 30000)) del i - def setUp(self): - """Start a ZEO server using a Unix domain socket - - The ZEO server uses the storage object returned by the - getStorage() method. - """ - self.running = 1 - self.__fs_base = tempfile.mktemp() - port = self.ports[0] - del self.ports[0] - self.addr = '', port - pid, exit = self._startServer() - self._pid = pid - self._server = exit - self.__super_setUp() - - def _startServer(self, create=1): - fs = FileStorage(self.__fs_base, create=create) - return forker.start_zeo_server(fs, self.addr) - def openClientStorage(self, cache='', cache_size=200000, wait=1): - base = ZEO.ClientStorage.ClientStorage(self.addr, - client=cache, - cache_size=cache_size, - wait_for_server_on_startup=wait) - storage = PackWaitWrapper(base) - storage.registerDB(DummyDB(), None) - return storage + # defined by subclasses + pass def shutdownServer(self): - if self.running: - self.running = 0 - self._server.close() - os.waitpid(self._pid, 0) + # defined by subclasses + pass def tearDown(self): """Try to cause the tests to halt""" @@ -277,15 +298,9 @@ revid1 = self._dostore(oid, data=obj) self.shutdownServer() self.running = 1 - self._pid, self._server = self._startServer(create=0) + self._startServer(create=0) oid = self._storage.new_oid() obj = MinPO(12) - # XXX This is a bloody pain. We're placing a heavy burden on - # users to catch a plethora of exceptions in order to write - # robust code. Need to think about implementing John Heintz's - # suggestion to make sure all exceptions inherit from - # POSException. - while 1: try: revid1 = self._dostore(oid, data=obj) @@ -294,6 +309,86 @@ time.sleep(0.1) else: break + # XXX This is a bloody pain. We're placing a heavy burden + # on users to catch a plethora of exceptions in order to + # write robust code. Need to think about implementing + # John Heintz's suggestion to make sure all exceptions + # inherit from POSException. + +class UnixConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + """Start a ZEO server using a Unix domain socket + + The ZEO server uses the storage object returned by the + getStorage() method. + """ + self.running = 1 + self.__fs_base = tempfile.mktemp() + self.addr = '', self.ports.pop() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + fs = FileStorage(self.__fs_base, create=create) + self._pid, self._server = forker.start_zeo_server(fs, self.addr) + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + self._server.close() + os.waitpid(self._pid, 0) + +class WindowsConnectionTests(ConnectionTests): + __super_setUp = StorageTestBase.StorageTestBase.setUp + + def setUp(self): + self.file = tempfile.mktemp() + self._startServer() + self.__super_setUp() + + def _startServer(self, create=1): + if create == 0: + port = self.addr[1] + else: + port = None + self.addr, self.test_a, pid = forker.start_zeo_server('FileStorage', + (self.file, + str(create)), + port) + self.running = 1 + + def openClientStorage(self, cache='', cache_size=200000, wait=1): + base = ZEO.ClientStorage.ClientStorage(self.addr, + client=cache, + cache_size=cache_size, + debug=1, + wait_for_server_on_startup=wait) + storage = PackWaitWrapper(base) + storage.registerDB(DummyDB(), None) + return storage + + def shutdownServer(self): + if self.running: + self.running = 0 + s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + s.connect(self.test_a) + s.close() + time.sleep(1.0) + + def tearDown(self): + self.shutdownServer() + def get_methods(klass): l = [klass] @@ -307,11 +402,18 @@ meth[k] = 1 return meth.keys() +if os.name == "posix": + test_classes = ZEOFileStorageTests, UnixConnectionTests +elif os.name == "nt": + test_classes = WindowsZEOFileStorageTests, WindowsConnectionTests +else: + raise RuntimeError, "unsupported os: %s" % os.name + def makeTestSuite(testname=''): suite = unittest.TestSuite() name = 'check' + testname lname = len(name) - for klass in ZEOFileStorageTests, ConnectionTests: + for klass in test_classes: for meth in get_methods(klass): if meth[:lname] == name: suite.addTest(klass(meth)) From jeremy at zope.com Thu Aug 23 21:50:58 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.9.2.4 Message-ID: <200108240150.f7O1owr27799@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv27778 Modified Files: Tag: zeo-1_0-branch testZEO.py Log Message: Merge changes from trunk (for Windows test success) With luck, all the necessary changes are included this time === ZEO/ZEO/tests/testZEO.py 1.9.2.3 => 1.9.2.4 === """ - __super_setUp = StorageTestBase.StorageTestBase.setUp + __super_tearDown = StorageTestBase.StorageTestBase.tearDown ports = [] for i in range(200): @@ -242,7 +242,7 @@ self.shutdownServer() # file storage appears to create four files for ext in '', '.index', '.lock', '.tmp': - path = self.__fs_base + ext + path = self.file + ext if os.path.exists(path): os.unlink(path) for i in 0, 1: @@ -325,13 +325,13 @@ getStorage() method. """ self.running = 1 - self.__fs_base = tempfile.mktemp() + self.file = tempfile.mktemp() self.addr = '', self.ports.pop() self._startServer() self.__super_setUp() def _startServer(self, create=1): - fs = FileStorage(self.__fs_base, create=create) + fs = FileStorage(self.file, create=create) self._pid, self._server = forker.start_zeo_server(fs, self.addr) def openClientStorage(self, cache='', cache_size=200000, wait=1): From jeremy at zope.com Thu Aug 23 21:50:58 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO/tests - testZEO.py:1.9.2.4 Message-ID: <200108240150.f7O1owP27804@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv27778 Modified Files: Tag: zeo-1_0-branch testZEO.py Log Message: Merge changes from trunk (for Windows test success) With luck, all the necessary changes are included this time === ZEO/ZEO/tests/testZEO.py 1.9.2.3 => 1.9.2.4 === """ - __super_setUp = StorageTestBase.StorageTestBase.setUp + __super_tearDown = StorageTestBase.StorageTestBase.tearDown ports = [] for i in range(200): @@ -242,7 +242,7 @@ self.shutdownServer() # file storage appears to create four files for ext in '', '.index', '.lock', '.tmp': - path = self.__fs_base + ext + path = self.file + ext if os.path.exists(path): os.unlink(path) for i in 0, 1: @@ -325,13 +325,13 @@ getStorage() method. """ self.running = 1 - self.__fs_base = tempfile.mktemp() + self.file = tempfile.mktemp() self.addr = '', self.ports.pop() self._startServer() self.__super_setUp() def _startServer(self, create=1): - fs = FileStorage(self.__fs_base, create=create) + fs = FileStorage(self.file, create=create) self._pid, self._server = forker.start_zeo_server(fs, self.addr) def openClientStorage(self, cache='', cache_size=200000, wait=1): From jeremy at zope.com Thu Aug 23 22:14:31 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientCache.py:1.16.2.1 Message-ID: <200108240214.f7O2EV028845@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv28787 Modified Files: Tag: zeo-1_0-branch ClientCache.py Log Message: Ignore errors removing files that don't exist. (Last missed change merged from the trunk, I think) === StandaloneZODB/ZEO/ClientCache.py 1.16 => 1.16.2.1 === def close(self): - self._f[self._current].close() + try: + self._f[self._current].close() + except OSError: + pass def open(self): self._acquire()