From jeremy at zope.com Tue Oct 2 19:16:03 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - StorageServer.py:1.28.2.4 zrpc.py:1.18.2.3 Message-ID: <200110022316.f92NG3329186@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv29131 Modified Files: Tag: zeo-1_0-branch StorageServer.py zrpc.py Log Message: Be careful about memory overhead of logging! The various logging calls did a lot of string formatting work that could consume very large amounts of memory. The worst case analysis is a store() call with a multi-megabyte pickle: We created a repr of the pickle then performed several string operations that copied the repr. The new code goes out of its way to avoid copying large strings. Regardless: ZEO should be substantially faster when run under python -O. === StandaloneZODB/ZEO/StorageServer.py 1.28.2.3 => 1.28.2.4 === from ZEO import trigger from ZEO import asyncwrap +from types import StringType class StorageServerError(POSException.StorageError): pass max_blather=120 def blather(*args): - m=string.join(map(str,args)) - if len(m) > max_blather: m=m[:max_blather]+' ...' + accum = [] + total_len = 0 + for arg in args: + if not isinstance(arg, StringType): + arg = str(arg) + accum.append(arg) + total_len = total_len + len(arg) + if total_len >= max_blather: + break + m = string.join(accum) + if len(m) > max_blather: m = m[:max_blather] + ' ...' LOG('ZEO Server', TRACE, m) @@ -258,7 +268,12 @@ def message_input(self, message, dump=dump, Unpickler=Unpickler, StringIO=StringIO, None=None): - if __debug__: blather('message_input', id(self), `message`) + if __debug__: + if len(message) > max_blather: + tmp = `message[:max_blather]` + else: + tmp = `message` + blather('message_input', id(self), tmp) if self.__storage is None: # This is the first communication from the client @@ -276,7 +291,9 @@ args=unpickler.load() name, args = args[0], args[1:] - if __debug__: blather('call %s: %s%s' % (id(self), name, `args`)) + if __debug__: + apply(blather, + ("call", id(self), ":", name,) + args) if not storage_method(name): raise 'Invalid Method Name', name @@ -294,7 +311,8 @@ self.return_error(sys.exc_info()[0], sys.exc_info()[1]) return - if __debug__: blather("%s R: %s" % (id(self), `r`)) + if __debug__: + blather("%s R: %s" % (id(self), `r`)) r=dump(r,1) self.message_output('R'+r) @@ -303,7 +321,8 @@ if type(err_value) is not type(self): err_value = err_type, err_value - if __debug__: blather("%s E: %s" % (id(self), `err_value`)) + if __debug__: + blather("%s E: %s" % (id(self), `err_value`)) try: r=dump(err_value, 1) except: === StandaloneZODB/ZEO/zrpc.py 1.18.2.2 => 1.18.2.3 === def message_input(self, m): if self._debug: - md=`m` - if len(m) > 60: md=md[:60]+' ...' + if len(m) > 60: + md = repr(m[:60]) + ' ...' + else: + md = repr(m) LOG(self._debug, TRACE, 'message_input %s' % md) c=m[:1] From jeremy at zope.com Wed Oct 3 08:58:08 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - zrpc.py:1.18.2.4 Message-ID: <200110031258.f93Cw8d05298@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv4351 Modified Files: Tag: zeo-1_0-branch zrpc.py Log Message: Add some explanatory comments to the exception propagation code. When an error occurs on the server, it sends an error response to the client. It's the client's responsibility to raise the exception contained in the error response to signal the error to the application. The traceback for this exception ends at the client's __call__() method, but the real error occurred somewhere in the server code. As a result, people are often confused by the tracebacks they see and don't know to look in the server's error log for the server-side traceback. Add a comment to this effect in the code and add a brief comment on the very line that raises the exception. The latter is useful because it will appear in the traceback the user will see. === StandaloneZODB/ZEO/zrpc.py 1.18.2.3 => 1.18.2.4 === if r=='RN.': return None # Common case! return loads(r[1:]) + + # If c == 'E', an error occured on the server. In + # this case, the return value is a pickled exception. + # Unpickle it and raise it on the client side. The + # traceback for this exception ends at this method, + # but the real error occurred somewhere in the server + # code. To diagnose the error, look for the real + # traceback in the server's zLOG output. if c=='E': try: r=loads(r[1:]) except: raise UnUnPickleableError(r[1:]) - if type(r) is TupleType: raise r[0], r[1] - raise r + if type(r) is TupleType: + raise r[0], r[1] # see server log for real traceback + raise r oob=self._outOfBand if oob is not None: r=r[1:] From jeremy at zope.com Thu Oct 4 18:43:11 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - testZEO.py:1.15 Message-ID: <200110042243.f94MhBU15009@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv14178/ZEO/tests Modified Files: testZEO.py Log Message: These storages all support revisions, so mixin RevisionStorage to the tests. === StandaloneZODB/ZEO/tests/testZEO.py 1.14 => 1.15 === from ZODB.tests import StorageTestBase, BasicStorage, VersionStorage, \ TransactionalUndoStorage, TransactionalUndoVersionStorage, \ - PackableStorage, Synchronization, ConflictResolution + PackableStorage, Synchronization, ConflictResolution, RevisionStorage from ZODB.tests.MinPO import MinPO from ZODB.tests.StorageTestBase import zodb_unpickle @@ -104,6 +104,7 @@ Cache.TransUndoStorageWithCache, BasicStorage.BasicStorage, VersionStorage.VersionStorage, + RevisionStorage.RevisionStorage, PackableStorage.PackableStorage, Synchronization.SynchronizedStorage, ConflictResolution.ConflictResolvingStorage, From jeremy at zope.com Fri Oct 19 14:59:35 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - StorageServer.py:1.28.2.5 Message-ID: <200110191859.f9JIxZI07927@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv7916 Modified Files: Tag: zeo-1_0-branch StorageServer.py Log Message: Make the pack_trigger an attribute of the server not the connection. There's no need to have one trigger for each connection. === StandaloneZODB/ZEO/StorageServer.py 1.28.2.4 => 1.28.2.5 === self.__get_connections=self.__connections.get + self._pack_trigger = trigger.trigger() asyncore.dispatcher.__init__(self) if type(connection) is type(''): @@ -245,7 +246,6 @@ self.__server=server self.__invalidated=[] self.__closed=None - self._pack_trigger = trigger.trigger() if __debug__: debug='ZEO Server' else: debug=0 SizedMessageAsyncConnection.__init__(self, sock, addr, debug=debug) @@ -415,11 +415,11 @@ error=sys.exc_info()) if wait: self.return_error(sys.exc_info()[0], sys.exc_info()[1]) - self._pack_trigger.pull_trigger() + self.__server._pack_trigger.pull_trigger() else: if wait: self.message_output('RN.') - self._pack_trigger.pull_trigger() + self.__server._pack_trigger.pull_trigger() else: # Broadcast new size statistics self.__server.invalidate(0, self.__storage_id, (), From jeremy at zope.com Fri Oct 19 14:59:00 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - trigger.py:1.1.8.3 Message-ID: <200110191859.f9JIx0b07836@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv7825 Modified Files: Tag: zeo-1_0-branch trigger.py Log Message: Add an __del__ that closes the pipe's file descriptors. === StandaloneZODB/ZEO/trigger.py 1.1.8.2 => 1.1.8.3 === def __init__ (self): - r, w = os.pipe() + r, w = self._fds = os.pipe() self.trigger = w asyncore.file_dispatcher.__init__ (self, r) self.lock = thread.allocate_lock() self.thunks = [] + + def __del__(self): + os.close(self._fds[0]) + os.close(self._fds[1]) def __repr__ (self): return '' % id(self) From jeremy at zope.com Mon Oct 22 19:44:58 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - forker.py:1.5.2.5 Message-ID: <200110222344.f9MNiw512783@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv12733 Modified Files: Tag: zeo-1_0-branch forker.py Log Message: Close the pipe on the client side. (The server's about to exit anyway, so it doesn't matter there.) === StandaloneZODB/ZEO/tests/forker.py 1.5.2.4 => 1.5.2.5 === def close(self): os.write(self.pipe, "done") + os.close(self.pipe) def start_zeo_server(storage, addr): rd, wr = os.pipe() From jeremy at zope.com Thu Oct 25 20:35:29 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - zrpc.py:1.18.2.5 Message-ID: <200110260035.f9Q0ZTP01156@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv1145 Modified Files: Tag: zeo-1_0-branch zrpc.py Log Message: spread out some code so it is easier to read === StandaloneZODB/ZEO/zrpc.py 1.18.2.4 => 1.18.2.5 === def finishConnect(self, s): - if self.__haveMainLoop: map=None # use the main loop map - else: map = {} # provide a dummy map + if self.__haveMainLoop: + map = None # use the main loop map + else: + map = {} # provide a dummy map SizedMessageAsyncConnection.__init__(self, s, '', map) # we are our own socket map! From jeremy at zope.com Thu Oct 25 20:35:52 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientStorage.py:1.33.2.2 Message-ID: <200110260035.f9Q0Zqt01183@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv1172 Modified Files: Tag: zeo-1_0-branch ClientStorage.py Log Message: Close the cache and temporary file when the storage closes. === StandaloneZODB/ZEO/ClientStorage.py 1.33.2.1 => 1.33.2.2 === # below. + def close(self): + self._cache.close() + self._tfile.close() + def registerDB(self, db, limit): """Register that the storage is controlled by the given DB. """ From jeremy at zope.com Thu Oct 25 20:36:31 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - forker.py:1.5.2.6 Message-ID: <200110260036.f9Q0aV901269@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv1258 Modified Files: Tag: zeo-1_0-branch forker.py Log Message: Make sure the socket is closed in get_port(). XXX Close the other end of the pipe in the child. Not sure if this affects the parent. === StandaloneZODB/ZEO/tests/forker.py 1.5.2.5 => 1.5.2.6 === s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) try: - s.connect(('localhost', port)) - except socket.error: - # XXX check value of error? - return port + try: + s.connect(('localhost', port)) + except socket.error: + # XXX check value of error? + return port + finally: + s.close() raise RuntimeError, "Can't find port" if os.name == "nt": @@ -98,6 +101,7 @@ ZEOServerExit(rd) serv = ZEO.StorageServer.StorageServer(addr, {'1':storage}) asyncore.loop() + os.close(rd) storage.close() if isinstance(addr, types.StringType): os.unlink(addr) From jeremy at zope.com Thu Oct 25 20:37:09 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - testZEO.py:1.9.2.5 Message-ID: <200110260037.f9Q0b9t01346@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv1335 Modified Files: Tag: zeo-1_0-branch testZEO.py Log Message: Make min_disconnect_poll smaller. Windows only: close the socket used to shutdown the server. === StandaloneZODB/ZEO/tests/testZEO.py 1.9.2.4 => 1.9.2.5 === forker.start_zeo_server(name, args) storage = ZEO.ClientStorage.ClientStorage(zeo_addr, debug=1, - min_disconnect_poll=0.5) + min_disconnect_poll=0.1) self._storage = PackWaitWrapper(storage) storage.registerDB(DummyDB(), None) @@ -194,6 +194,7 @@ self._storage.close() s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(self.test_addr) + s.close() # the connection should cause the storage server to die ## os.waitpid(self.test_pid, 0) time.sleep(0.5) From jeremy at zope.com Thu Oct 25 20:50:38 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientStorage.py:1.33.2.3 Message-ID: <200110260050.f9Q0ocs04430@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv4419 Modified Files: Tag: zeo-1_0-branch ClientStorage.py Log Message: Previous fix was all wrong! There was already a close() method. In the real close(), close the _tfile. === StandaloneZODB/ZEO/ClientStorage.py 1.33.2.2 => 1.33.2.3 === # below. - def close(self): - self._cache.close() - self._tfile.close() - def registerDB(self, db, limit): """Register that the storage is controlled by the given DB. """ @@ -312,6 +308,7 @@ LOG("ClientStorage", INFO, "close") self._call.closeIntensionally() self._cache.close() + self._tfile.close() self.closed = 1 finally: self._lock_release() From jeremy at zope.com Fri Oct 26 12:58:19 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - zrpc.py:1.18.2.6 Message-ID: <200110261658.f9QGwJj13539@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv13528 Modified Files: Tag: zeo-1_0-branch zrpc.py Log Message: When an asyncRPC object is closed, release reference to its out-of-band hook. The oob hook participates in a cycle of references to the Invalidator. === StandaloneZODB/ZEO/zrpc.py 1.18.2.5 => 1.18.2.6 === else: self.close() + self._outOfBand = None self.__closed = 1 def close(self): @@ -312,6 +313,8 @@ self.aq_parent.notifyDisconnected(self) # causes read call to raise last exception, which should be # the socket error that caused asyncore to close the socket. - self.__r='E'+dump(sys.exc_info()[:2], 1) - try: self.__lr() - except: pass + self.__r = 'E' + dump(sys.exc_info()[:2], 1) + try: + self.__lr() + except: + pass From jeremy at zope.com Fri Oct 26 13:20:38 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientCache.py:1.16.2.2 ClientStorage.py:1.33.2.4 Invalidator.py:1.5.2.1 Message-ID: <200110261720.f9QHKci17991@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv17975 Modified Files: Tag: zeo-1_0-branch ClientCache.py ClientStorage.py Invalidator.py Log Message: Eliminate leaks of Invalidator objects. Explicitly break cycles in the closeIntensionally() call on ClientStorage. Add a close() method to Invalidator that gives up its references to other objects. XXX We still leak ClientCache objects for reasons I can't fathom. We leak a ClientCache each time a ClientStorage object is created. I'm not overly concerned because applications will almost certainly create a single instance and not many. === StandaloneZODB/ZEO/ClientCache.py 1.16.2.1 => 1.16.2.2 === try: self._f[self._current].close() - except OSError: + except ValueError: pass def open(self): @@ -373,6 +373,8 @@ self._f[current]=open(self._p[current],'w+b') else: # Temporary cache file: + if self._f[current] is not None: + self._f[current].close() self._f[current] = tempfile.TemporaryFile(suffix='.zec') self._f[current].write(magic) self._pos=pos=4 === StandaloneZODB/ZEO/ClientStorage.py 1.33.2.3 => 1.33.2.4 === # called until after this call. - invalidator=Invalidator.Invalidator( - db.invalidate, - self._cache.invalidate) + self.invalidator = Invalidator.Invalidator(db.invalidate, + self._cache.invalidate) def out_of_band_hook( code, args, get_hook={ - 'b': (invalidator.begin, 0), - 'i': (invalidator.invalidate, 1), - 'e': (invalidator.end, 0), - 'I': (invalidator.Invalidate, 1), + 'b': (self.invalidator.begin, 0), + 'i': (self.invalidator.invalidate, 1), + 'e': (self.invalidator.end, 0), + 'I': (self.invalidator.Invalidate, 1), 'U': (self._commit_lock_release, 0), 's': (self._serials.append, 1), 'S': (self._info.update, 1), @@ -307,9 +306,12 @@ try: LOG("ClientStorage", INFO, "close") self._call.closeIntensionally() - self._cache.close() self._tfile.close() - self.closed = 1 + self._cache.close() + if self.invalidator is not None: + self.invalidator.close() + self.invalidator = None + self.closed = 1 finally: self._lock_release() def commitVersion(self, src, dest, transaction): @@ -318,7 +320,6 @@ self._lock_acquire() try: oids=self._call('commitVersion', src, dest, self._serial) - invalidate=self._cache.invalidate if dest: vlen = pack(">H", len(src)) # just invalidate our version data @@ -523,7 +524,6 @@ seek=tfile.seek read=tfile.read cache=self._cache - update=cache.update size=tfile.tell() cache.checkSize(size) seek(0) @@ -544,9 +544,9 @@ "temporary file." ) if s==ResolvedSerial: - cache.invalidate(oid, v) + self._cache.invalidate(oid, v) else: - update(oid, s, v, p) + self._cache.update(oid, s, v, p) i=i+15+vlen+dlen elif opcode == "i": oid=read(8) @@ -579,7 +579,8 @@ try: oids=self._call('undo', transaction_id) cinvalidate=self._cache.invalidate - for oid in oids: cinvalidate(oid,'') + for oid in oids: + cinvalidate(oid,'') return oids finally: self._lock_release() === StandaloneZODB/ZEO/Invalidator.py 1.5 => 1.5.2.1 === self.cinvalidate=cinvalidate + def close(self): + self.dinvalidate = None + self.cinvalidate = None + def begin(self): self._tfile=tempfile.TemporaryFile() pickler=cPickle.Pickler(self._tfile, 1) From jeremy at zope.com Fri Oct 26 13:30:32 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - stress.py:1.1.2.1 Message-ID: <200110261730.f9QHUWO20128@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv20117 Added Files: Tag: zeo-1_0-branch stress.py Log Message: Simple stress test that creates many clients to use a single server. === Added File StandaloneZODB/ZEO/tests/stress.py === """A ZEO client-server stress test to look for leaks. The stress test should run in an infinite loop and should involve multiple connections. """ from __future__ import nested_scopes import ZODB from ZEO.ClientStorage import ClientStorage from ZODB.MappingStorage import MappingStorage from ZEO.tests import forker from ZODB.tests import MinPO import zLOG import os import random import sys import types NUM_TRANSACTIONS_PER_CONN = 10 NUM_CONNECTIONS = 10 NUM_ROOTS = 20 MAX_DEPTH = 20 MIN_OBJSIZE = 128 MAX_OBJSIZE = 2048 def an_object(): """Return an object suitable for a PersistentMapping key""" size = random.randrange(MIN_OBJSIZE, MAX_OBJSIZE) if os.path.exists("/dev/urandom"): f = open("/dev/urandom") buf = f.read(size) f.close() return buf else: f = open(MinPO.__file__) l = list(f.read(size)) f.close() random.shuffle(l) return "".join(l) def setup(cn): """Initialize the database with some objects""" root = cn.root() for i in range(NUM_ROOTS): prev = an_object() for j in range(random.randrange(1, MAX_DEPTH)): o = MinPO.MinPO(prev) prev = o root[an_object()] = o get_transaction().commit() cn.close() def work(cn): """Do some work with a transaction""" cn.sync() root = cn.root() obj = random.choice(root.values()) # walk down to the bottom while not isinstance(obj.value, types.StringType): obj = obj.value obj.value = an_object() get_transaction().commit() def main(): # Yuck! Need to cleanup forker so that the API is consistent # across Unix and Windows, at least if that's possible. if os.name == "nt": zaddr, tport, pid = forker.start_zeo_server('MappingStorage', ()) def exitserver(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.connect(tport) s.close() else: zaddr = '', random.randrange(20000, 30000) pid, exitobj = forker.start_zeo_server(MappingStorage(), zaddr) def exitserver(): exitobj.close() while 1: pid = start_child(zaddr) print "started", pid os.waitpid(pid, 0) exitserver() def start_child(zaddr): pid = os.fork() if pid != 0: return pid storage = ClientStorage(zaddr, debug=1, min_disconnect_poll=0.5) db = ZODB.DB(storage, pool_size=NUM_CONNECTIONS) setup(db.open()) conns = [] conn_count = 0 for i in range(NUM_CONNECTIONS): c = db.open() c.__count = 0 conns.append(c) conn_count += 1 while conn_count < 25: c = random.choice(conns) if c.__count > NUM_TRANSACTIONS_PER_CONN: conns.remove(c) c.close() conn_count += 1 c = db.open() c.__count = 0 conns.append(c) else: c.__count += 1 work(c) os._exit(0) if __name__ == "__main__": main() From jeremy at zope.com Fri Oct 26 15:34:29 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientStorage.py:1.33.2.5 Message-ID: <200110261934.f9QJYTr14638@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv14627 Modified Files: Tag: zeo-1_0-branch ClientStorage.py Log Message: Make supportsTransactionalUndo() robust in the face of a storage server that doesn't even know about transactional undo. === StandaloneZODB/ZEO/ClientStorage.py 1.33.2.4 => 1.33.2.5 === finally: self._lock_release() - - - def supportsUndo(self): return self._info['supportsUndo'] - def supportsVersions(self): return self._info['supportsVersions'] + def supportsUndo(self): + return self._info['supportsUndo'] + + def supportsVersions(self): + return self._info['supportsVersions'] + def supportsTransactionalUndo(self): - return self._info['supportsTransactionalUndo'] + try: + return self._info['supportsTransactionalUndo'] + except KeyError: + return 0 def tpc_abort(self, transaction): self._lock_acquire() From jeremy at zope.com Fri Oct 26 15:37:04 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - Cache.py:1.3.2.1 testZEO.py:1.9.2.6 Message-ID: <200110261937.f9QJb4F14815@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv14802 Modified Files: Tag: zeo-1_0-branch Cache.py testZEO.py Log Message: Update tests so that ZEO tests pass on Zope 2.3. There are a few problems with Zope 2.3 that prevented the tests from succeeding. Since the 1.0 release must support Zope 2.3, we'll need to hack the various tests to work around them. Assuming releases after 1.0 don't need to work with 2.3: After the 1.0 final release, I'll tag both ZEO and ZODB trees and then undo all these silly changes. Problems: - no support for transactional undo or getSerial() - FileStorage commitVersion() and abortVersion() didn't check for bogus arguments === StandaloneZODB/ZEO/tests/Cache.py 1.3 => 1.3.2.1 === info = self._storage.undoInfo() + if not info: + # XXX perhaps we have an old storage implementation that + # does do the negative nonsense + info = self._storage.undoInfo(0, 20) tid = info[0]['id'] + + # We may need to bail at this point if the storage doesn't + # support transactional undo + if not self._storage.supportsTransactionalUndo(): + return + # Now start an undo transaction self._transaction.note('undo1') self._storage.tpc_begin(self._transaction) === StandaloneZODB/ZEO/tests/testZEO.py 1.9.2.5 => 1.9.2.6 === d[oid] = serial return d + +# Some of the ZEO tests depend on the version of FileStorage available +# for the tests. If we run these tests using Zope 2.3, FileStorage +# doesn't support TransactionalUndo. + +if hasattr(FileStorage, 'supportsTransactionalUndo'): + # XXX Assume that a FileStorage that supports transactional undo + # also supports conflict resolution. + class VersionDependentTests( + TransactionalUndoStorage.TransactionalUndoStorage, + TransactionalUndoVersionStorage.TransactionalUndoVersionStorage, + ConflictResolution.ConflictResolvingStorage, + ConflictResolution.ConflictResolvingTransUndoStorage): + pass +else: + class VersionDependentTests: + pass class GenericTests(ZEOTestBase, + VersionDependentTests, Cache.StorageWithCache, Cache.TransUndoStorageWithCache, BasicStorage.BasicStorage, VersionStorage.VersionStorage, PackableStorage.PackableStorage, Synchronization.SynchronizedStorage, - ConflictResolution.ConflictResolvingStorage, - ConflictResolution.ConflictResolvingTransUndoStorage, - TransactionalUndoStorage.TransactionalUndoStorage, - TransactionalUndoVersionStorage.TransactionalUndoVersionStorage, ): """An abstract base class for ZEO tests From jeremy at zope.com Mon Oct 29 16:40:22 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - ClientCache.py:1.16.2.3 ClientStorage.py:1.33.2.6 Message-ID: <200110292140.f9TLeMd18328@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv18014 Modified Files: Tag: zeo-1_0-branch ClientCache.py ClientStorage.py Log Message: Be careful when closeing a tempfile. On Windows, multiple calls to close() will raise an exception. On Unix, the extra calls are ignored. === StandaloneZODB/ZEO/ClientCache.py 1.16.2.2 => 1.16.2.3 === try: self._f[self._current].close() - except ValueError: + except (os.error, ValueError): pass def open(self): === StandaloneZODB/ZEO/ClientStorage.py 1.33.2.5 => 1.33.2.6 === LOG("ClientStorage", INFO, "close") self._call.closeIntensionally() - self._tfile.close() + try: + self._tfile.close() + except os.error: + # On Windows, this can fail if it is called more than + # once, because it tries to delete the file each + # time. + pass self._cache.close() if self.invalidator is not None: self.invalidator.close() From jeremy at zope.com Mon Oct 29 22:08:54 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO/tests - forker.py:1.5.2.7 Message-ID: <200110300308.f9U38sX23175@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO/tests In directory cvs.zope.org:/tmp/cvs-serv23135 Modified Files: Tag: zeo-1_0-branch forker.py Log Message: Don't get a new port if one is specified as an argument. === StandaloneZODB/ZEO/tests/forker.py 1.5.2.6 => 1.5.2.7 === """ import ZEO.tests.winserver - port = get_port() + if port is None: + port = get_port() script = ZEO.tests.winserver.__file__ if script.endswith('.pyc'): script = script[:-1] From jeremy at zope.com Tue Oct 30 16:35:48 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO - CHANGES.txt:1.25.2.5 Message-ID: <200110302135.f9ULZmA26031@cvs.baymountain.com> Update of /cvs-repository/ZEO In directory cvs.zope.org:/tmp/cvs-serv26020 Modified Files: Tag: zeo-1_0-branch CHANGES.txt Log Message: Add bug fix list for 1.0b5 === ZEO/CHANGES.txt 1.25.2.4 => 1.25.2.5 === + ZEO 1.0 beta 5 + + Bugs fixed + + - Fix a bug introduced in earlier beta that caused the storage + server to leak two file descriptors per client connection, + eventually leading to a crash. + + - Reduced the amount of memory consumed by the logging + infrastructure. + + - There were numerous small changes to close() methods and the + test suite to avoid leaking (much) memory when a single process + opens and closes many ClientStorage objects. + ZEO 1.0 beta 4 Bugs fixed From jeremy at zope.com Wed Oct 31 14:53:22 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: StandaloneZODB/ZEO - StorageServer.py:1.28.2.6 Message-ID: <200110311953.f9VJrMo02922@cvs.baymountain.com> Update of /cvs-repository/StandaloneZODB/ZEO In directory cvs.zope.org:/tmp/cvs-serv2911/ZEO Modified Files: Tag: zeo-1_0-branch StorageServer.py Log Message: Fix __main__ so it actually works. Reformat a for statement so the body of the loop is on a separate line. XXX Neither of these changes requires a new beta release. === StandaloneZODB/ZEO/StorageServer.py 1.28.2.5 => 1.28.2.6 === self.__storages=storages - for n, s in storages.items(): init_storage(s) + for n, s in storages.items(): + init_storage(s) self.__connections={} self.__get_connections=self.__connections.get @@ -601,6 +602,7 @@ port='', int(port) except: pass - - StorageServer(port, ZODB.FileStorage.FileStorage(name)) + + d = {'1': ZODB.FileStorage.FileStorage(name)} + StorageServer(port, d) asyncwrap.loop() From jeremy at zope.com Wed Oct 31 15:03:03 2001 From: jeremy at zope.com (Jeremy Hylton) Date: Sun Aug 10 16:31:16 2008 Subject: [ZEO-Checkins] CVS: ZEO/ZEO - start.py:1.24.2.1 Message-ID: <200110312003.f9VK33303574@cvs.baymountain.com> Update of /cvs-repository/ZEO/ZEO In directory cvs.zope.org:/tmp/cvs-serv3563 Modified Files: Tag: zeo-1_0-branch start.py Log Message: Replace another "finally: pass" with "except: pass" XXX A more thorough solution would be to log an error message when something goes wrong closing the storage. === ZEO/ZEO/start.py 1.24 => 1.24.2.1 === for storage in storages.values(): try: storage.close() - finally: pass + except: pass try: from zLOG import LOG, INFO