[ZODB-Dev] Errors running a Stress Test

[Brandon Ehle]

Here is the same setup but running with the same test with the ZODB
module at HEAD (and fixing that startup db bug you found).

You wanted all the exceptions that I did not post last time so here goes. :)

*****
I get one of these at startup (server side) when the DB is being
created.  Is this just an error related to fact that it's looking for
the database before it exists?

Traceback (most recent call last):
 File "C:\Python23\Lib\site-packages\ZEO\zrpc\connection.py", line 266,
in handle_request
   ret = meth(*args)
 File "C:\Python23\Lib\site-packages\ZEO\StorageServer.py", line 238,
in loadEx
   return self.storage.loadEx(oid, version)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\FileStorage.py",
line 529, in loadEx
   pos = self._lookup_pos(oid)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\FileStorage.py",
line 520, in _lookup_pos
   raise POSKeyError(oid)
POSKeyError: 0000000000000000

*****
Running with 1 thread I got one of these once.  This error seemed to
hang the client.  After restarting the client, no commit() operations
succeeded until I deleted the client cache.

Exception in thread Thread-1:
Traceback (most recent call last):
 File "C:\Python23\lib\threading.py", line 436, in __bootstrap
   self.run()
 File "C:\Python23\lib\threading.py", line 416, in run
   self.__target(*self.__args, **self.__kwargs)
 File "stress.py", line 48, in hammer_the_db
   random.choice([commit] + [abort])()
 File "stress.py", line 25, in commit
   get_transaction().commit()
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 257, in
commit
   self._cleanup(objects, ncommitted, jars, subjars)
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 253, in
commit
   self._finish_one(jars[0])
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 362, in
_finish_one
   jar.tpc_finish(self)
 File "C:\Python23\Lib\site-packages\ZODB\Connection.py", line 917, in
tpc_finish
   self._storage.tpc_finish(transaction, callback)
 File "C:\Python23\Lib\site-packages\ZEO\ClientStorage.py", line 961,
in tpc_finish
   self._update_cache(tid)
 File "C:\Python23\Lib\site-packages\ZEO\ClientStorage.py", line 993,
in _update_cache
   self._cache.store(oid, version, s, None, data)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 274, in store
   self.fc.add(o)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 779, in add
   available = self._makeroom(size)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 725, in _makeroom
   self._evictobj(e, size)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 814, in _evictobj
   self.parent._evicted(o)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 362, in _evicted
   del self.current[oid]
KeyError: '\x00\x00\x00\x00\x00\x00\x00\x00'

*****
Running with 10 threads I got lots of these client side:

Exception in thread Thread-1:
Traceback (most recent call last):
 File "C:\Python23\lib\threading.py", line 436, in __bootstrap
   self.run()
 File "C:\Python23\lib\threading.py", line 416, in run
   self.__target(*self.__args, **self.__kwargs)
 File "stress.py", line 48, in hammer_the_db
   random.choice([commit] + [abort])()
 File "stress.py", line 25, in commit
   get_transaction().commit()
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 257, in
commit
   self._cleanup(objects, ncommitted, jars, subjars)
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 253, in
commit
   self._finish_one(jars[0])
 File "C:\Python23\Lib\site-packages\ZODB\Transaction.py", line 362, in
_finish_one
   jar.tpc_finish(self)
 File "C:\Python23\Lib\site-packages\ZODB\Connection.py", line 917, in
tpc_finish
   self._storage.tpc_finish(transaction, callback)
 File "C:\Python23\Lib\site-packages\ZEO\ClientStorage.py", line 961,
in tpc_finish
   self._update_cache(tid)
 File "C:\Python23\Lib\site-packages\ZEO\ClientStorage.py", line 993,
in _update_cache
   self._cache.store(oid, version, s, None, data)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 274, in store
   self.fc.add(o)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 779, in add
   available = self._makeroom(size)
 File "C:\Python23\Lib\site-packages\ZEO\cache.py", line 723, in _makeroom
   size, e = self.filemap.pop(ofs)
KeyError: 12

*****
Running with more than many threads (10+) I get lots of these server side.

2004-03-16T13:15:20 ERROR(200) Conflict Resolution Unexpected error
Traceback (most recent call last):
 File "C:\Python23\Lib\site-packages\ZODB\ConflictResolution.py", line
120, in tryToResolveConflict
   old = state(self, oid, oldSerial, prfactory)
 File "C:\Python23\Lib\site-packages\ZODB\ConflictResolution.py", line
51, in state
   p = p or self.loadSerial(oid, serial)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\FileStorage.py",
line 580, in loadSerial
   raise POSKeyError(oid)
POSKeyError: 0000000000000001

*****
A server side errro caused by running two instances of the app on the
same machine.  My aggresive packing seems to be causing this (but it is
needed or I quickly run out of disk space).

2004-03-16T13:17:17 INFO(0) ZSS:3284/10.14.243.141:4561
pack(time=1079471837.4430001) started...
------
2004-03-16T13:17:17 ERROR(200) zrpc:3284 Error raised in delayed method
Traceback (most recent call last):
 File "C:\Python23\Lib\site-packages\ZEO\StorageServer.py", line 1001,
in run
   result = self._method(*self._args)
 File "C:\Python23\Lib\site-packages\ZEO\StorageServer.py", line 335,
in _pack_impl
   self.storage.pack(time, referencesf)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\FileStorage.py",
line 1333, in pack
   opos = p.pack()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
464, in pack
   self.gc.findReachable()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
218, in findReachable
   self.buildPackIndex()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
278, in buildPackIndex
   raise FileStorageError(
FileStorageError: The database has already been packed to a later time
or no changes have been made since the last pack

*****
Another server side error caused by two instance of the app running on
the same machine and trying pack at the same time.  I probably could
store some state in the DB to limit only one instance to be able to pack
at a time or use a seperate process for packing.  My major complaint
with this is that to trap this error you have to catch the class
CorruptedError(), but I want real corrupted errors to be fatal (maybe
this is?)

2004-03-16T13:17:57 INFO(0) ZSS:3284/10.14.243.141:4563
pack(time=1079471869.911) started...
------
2004-03-16T13:17:57 ERROR(200) FS pack zodb.fs:609:transaction with
checkpoint flag set
------
2004-03-16T13:17:57 ERROR(200) zrpc:3284 Error raised in delayed method
Traceback (most recent call last):
 File "C:\Python23\Lib\site-packages\ZEO\StorageServer.py", line 1001,
in run
   result = self._method(*self._args)
 File "C:\Python23\Lib\site-packages\ZEO\StorageServer.py", line 335,
in _pack_impl
   self.storage.pack(time, referencesf)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\FileStorage.py",
line 1333, in pack
   opos = p.pack()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
464, in pack
   self.gc.findReachable()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
220, in findReachable
   self.findReachableFromFuture()
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\fspack.py", line
321, in findReachableFromFuture
   self.checkTxn(th, pos)
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\format.py", line
256, in checkTxn
   self.fail(pos, "transaction with checkpoint flag set")
 File "C:\Python23\Lib\site-packages\ZODB\FileStorage\format.py", line
248, in fail
   raise CorruptedError(s)
CorruptedError: zodb.fs:609:transaction with checkpoint flag set

*****
Got one of these client side at the same time.

Traceback (most recent call last):
 File "stress.py", line 73, in ?
   db.pack()
 File "C:\Python23\Lib\site-packages\ZODB\DB.py", line 581, in pack
   self._storage.pack(t, referencesf)
 File "C:\Python23\Lib\site-packages\ZEO\ClientStorage.py", line 853,
in pack
   return self._server.pack(t, wait)
 File "C:\Python23\Lib\site-packages\ZEO\ServerStub.py", line 161, in pack
   self.rpc.call('pack', t, wait)
 File "C:\Python23\Lib\site-packages\ZEO\zrpc\connection.py", line 374,
in call
   raise inst # error raised by server
CorruptedError: zodb.fs:7687:transaction with checkpoint flag set