[ZODB-Dev] AssertionError on multithread ZEO app

[Pedro Werneck]

Hi all


I'm writing a multithread app using ZODB/ZEO for persistence. The app
consists in a main app which start the connection with ZEO with
ClientStorage, launch several threads to gather, parse and fill the DB
with data from the network, and, just for testing purposes now, a loop
printing the data in the DB every 5 seconds. Each parser store its data
in a PersistentList in conn.dbroot() and commit the changes.

The problem is... I can't see the committed changes from the main
thread, unless I do a commit there too, which seems strange. Not only
that, but sometimes I get an assertion error on ZEO/cache.py.

This is the first time I'm using ZODB/ZEO for multithreaded app, so
maybe I'm doing something really stupid and did not notice.

Another problem not related to this is the size and how fast the DB
grows. I'm think this is related to undo features, so I would like to
know if it's possible to disable it or something else.

These are the relevant parts of my code. The code for commiting data on
each thread:

###
        for x in xrange(self._commit_attempts):
            try:
                del self.reg_db[:]
                self.reg_db.extend(regs)
                transaction.get().commit()
            except ConflictError, msg:
                self.logger.debug(
                    'ConflictError:%s pause a while and retry'%msg
                    )
                transaction.get().abort()
                time.sleep(0.5)
            else:
                logger.debug('Commited succesfully')
                return

        logger.critical('Commit failed')
###            

The test loop:

###
    while 1:
        for name, regs in dbroot.items():
            logger.debug("DB list for '%s' with %s regs"%(name,
len(regs)))
        time.sleep(5)

        # I only get updates with this commit here
        transaction.get().commit() 

###

This is the traceback for the assertion error (line 86 is the commit
mentioned above):

Traceback (most recent call last):
  File "test_runparsers.py", line 48, in ?
    main()
  File "../../runparsers.py", line 86, in main
    transaction.get().commit()
  File "/usr/lib/python2.4/site-packages/transaction/_transaction.py",
line 380, in commit
    self._saveCommitishError() # This raises!
  File "/usr/lib/python2.4/site-packages/transaction/_transaction.py",
line 378, in commit
    self._commitResources()
  File "/usr/lib/python2.4/site-packages/transaction/_transaction.py",
line 441, in _commitResources
    rm.tpc_finish(self)
  File "/usr/lib/python2.4/site-packages/ZODB/Connection.py", line 679,
in tpc_finish
    self._storage.tpc_finish(transaction, callback)
  File "/usr/lib/python2.4/site-packages/ZEO/ClientStorage.py", line
974, in tpc_finish
    self._update_cache(tid)
  File "/usr/lib/python2.4/site-packages/ZEO/ClientStorage.py", line
999, in _update_cache
    self._cache.invalidate(oid, version, tid)
  File "/usr/lib/python2.4/site-packages/ZEO/cache.py", line 367, in
invalidate
    assert tid is not None and cur_tid < tid
AssertionError

And the traceback for the exception ocurred at the server when I get 
the assertion error on the app

Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/ZEO/zrpc/connection.py", line
421, in handle_request
    ret = meth(*args)
  File "/usr/lib/python2.4/site-packages/ZEO/StorageServer.py", line
248, in loadEx
    return self.storage.loadEx(oid, version)
  File
"/usr/lib/python2.4/site-packages/ZODB/FileStorage/FileStorage.py", line
523, in loadEx
    pos = self._lookup_pos(oid)
  File
"/usr/lib/python2.4/site-packages/ZODB/FileStorage/FileStorage.py", line
514, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x00


Thanks for any help

-- 
Pedro Werneck